Is anecdotal evidence dangerous?

[Doc]

I have read so many posts that use only anecdotal evidence to support a particular point of view.

Man, if you think this is bad, go try scubadiving.com Park your analytical brain at the door.

 

[Charlie99]

Scientists will fudge their own data (there have been many instances of photoshoped data, even). Scientists will try to salvage a failed project by twisting the data as much as possible and finding correlations just to publish SOMETHING, because their careers depend on it. Science aims to be logical, but still encounters the same human pitfalls.

Fudging and ill intent aren't required for there to be problems.

An even more common pitfall is where you already know the "correct' answer that your instruments should be telling you.
If your results of a test/measurement don't quite match up with what everyone else has measured, there must have been some minor disturbance or side effect that you didn't account for. So you go looking for it, and correct the problem.

You run the tests again and still don't get quite the accepted answer. So you repeat the process looking for additional potential problems in your experimental setup.

When you finally get the "right" answer, you are satisfied and stop looking for additional error sources. Of course, they may still be problems.

No bad intentions involved, yet multiple supposedly independent investigators come up with the same wrong answer. (Yep. I've guilty of the above). This sort of problem is most common when the measurements are difficult/expensive and noisy (as in not very repeatable), and change as the result of multiple influences in addition to the one you are trying to measure the effect of. That describes a lot of real world problems, including decompression.

Charlie Allen

 

[everlasting]

Fudging and ill intent aren't required for there to be problems.

An even more common pitfall is where you already know the "correct' answer that your instruments should be telling you.
If your results of a test/measurement don't quite match up with what everyone else has measured, there must have been some minor disturbance or side effect that you didn't account for. So you go looking for it, and correct the problem.

Very true. I'm guilty of the above as well, and I'm glad you mentioned it. Although I wasn't looking at my examples as motivated by "ill intent" per se and more by desperation.

The important thing to remember is that a sample size of thousands may not necessarily be more accurate than a sample size of one and science is not always more trustworthy than anecdotal evidence.

 

[Cacia]

When you finally get the "right" answer, you are satisfied and stop looking for additional error sources. Of course, they may still be problems.

Isn't there a term for that, in research?

 

[Thalassamania]

I have rarely seen so little understanding of the scientific method and actual practices. When someone fudges their work (and it happens) they inevitably get caught when no one else can confirm their results.

 

[Cacia]

Not always, you have entire institutions invested in the process, and funding enters the picture too. Maybe theoretically, you are right.

You should read about the Women's Health Initiative and the Provera studies, Thall.

I have had quite a few UCLA trained researchers tell me how it was manipulated. There are many inferior studies being done in medical research, thanks to the pharmacuetical companies. Hell, nations get invested in outcomes.

The corruption is not even always overt, in fact, usually it is not.

You sure have a lot of faith in *them,* for a skeptic. Maybe science is your religion?

 

[everlasting]

Look, I'm not discounting science or saying that the scientific method isn't important and doesn't work and I don't want to discuss the frequency of fudging/misinterpretation of data and when and how it gets caught.

Even assuming that all fudged/misinterpreted data gets caught eventually, that doesn't mean that a non scientist didn't read and believe it before that happened and did something according to that false information. Or that a non scientist read a scientific paper with a study that was inherently flawed without knowing it, and believed it blindly. Or just that a non scientist misinterprets the results of a well thought out, accurate study. Or all the "non scientists" that I mentioned could be scientists (in a different or the same field).

My point is that science is not perfect. Data gets fudged. Data gets twisted. Scientists get so caught up and trying to get the answer that they want that they ignore the real answer. Mistakes are fixed, but that takes time. In the meantime, we have to examine scientific studies with a critical eye and remember that thousands of data points doesn't necessarily mean accurate information. And unless you're intimately familiar with the methods and standards of the particular field, which most people won't be, it's difficult to assess the validity of a study on your own. Whereas it's much easier to assess the validity of an anecdote when you have some prior knowledge about the subject.

 

[Thalassamania]

Please do not confuse medical research and science. While there are excellent medical researchers out there, physicians are not trained scientists, either practically or ethically and that lack of background shows in many areas.

 

[Cacia]

Point taken.

but they think they are.

 

[Thalassamania]

My point is that science is not perfect. Data gets fudged. Data gets twisted. Scientists get so caught up and trying to get the answer that they want that they ignore the real answer. Mistakes are fixed, but that takes time. In the meantime, we have to examine scientific studies with a critical eye and remember that thousands of data points doesn't necessarily mean accurate information. And unless you're intimately familiar with the methods and standards of the particular field, which most people won't be, it's difficult to assess the validity of a study on your own. Whereas it's much easier to assess the validity of an anecdote when you have some prior knowledge about the subject.

Science is not perfect and people are not perfect, if you live out on the edge of science you need to be prepared for some disappointments. If you are not, "intimately familiar with the methods and standards of the particular field," then I do not suggest that you attempt to participate at that level, go back a few rungs and leave the truth finding to the pros.