Studies studies

[ponkatart42]

Hello Dr. Bornfeld,

Why is it that when studies indicate something it isn't considered a proven fact?, especially consistantly, when it's more than one study.

For instanace, with Periodontal Disease, studies done over the paste 10 years have indicated that PD raises the risk of one getting all sorts of ailments. There must be some explanation in the medical world that we lay people dont understand with our elementary way of thinking. Thank you.

[mbornfeld]

Dear ponkatart42,

Your question touches on a complex array of different disciplines: philosophy, logic, statistics, and even theology. It also goes to the heart of the scientific process itself, which is founded on doubt and skepticism rather than faith and belief.

You are quite right: the conclusions drawn from a scientific study are not fact, and can never be construed as such. They may be dumbed-down and misrepresented as fact in the mass media, but research looks for clues, insights, correlations, and relationships. But fact? No. The great philosophers have debated truth and fact, and man's ability to distinguish fact from fiction. The only universal truth that anyone can believe is that the real truth cannot be unequivocally known-- outside the world of religion.

There are several principles that apply when reading a published scientific study:

• Check that the study has collected a representative sample of test subjects. For example, many studies are run at college campuses, and they typically recruit students as test subjects. Depending on the school, that fact alone biases the selection process; the demographics will be skewed along age, ethnic, sex, and socioeconomic lines. Any conclusions drawn from that study might have limited applicability to individuals who don't closely match the participant demographic
• Check that the study has a large number of test subjects. This is a commonly violated principle, because it is often so difficult to recruit study participants. Nonetheless, a perfunctory reading of an abstract of such a study might be interpreted as providing conclusive information, when it is anything but.
• Check the statistical model used. I can't help you here-- I've never liked statistics, but they are a blunt or sharp tool that has evoked many witty aphorisms. My favorite is: "Some people use statistics the same way a drunk uses a lamp post-- for support rather than illumination". Again, if you are the type that skips right over the entire publication to the "conclusions" section, you're apt to miss an opportunity to catch a flaw in its reasoning.
• Check that it is a double blind study. This means that the study employs an experimental group of subjects, a control group of subjects, and neither the experimenter nor the subject knows to which group each subject belongs. This essential study configuration is intended to eliminate experimenter bias, which is a real issue. Almost any experimenter either wants the study to prove something, or believes the study will prove or disprove something. Either way, desires or expectations have a way of influencing how experimental data is collected, compiled, and interpreted-- a bias known as the "Rosenthal effect" after psychologist Robert Rosenthal.
• Check the "researcher disclosures" section. I've saved the best for last, because it is the most potentially problematic. In many cases (and I mean MANY cases), a research study is biased due to a financial stake in the results. To illustrate this point, you may have heard of a recent study that found that persons who consume two or more cups of coffee a day have a statistically lower risk of developing Alzheimer's disease. One of the researchers was Juan Valdez. Well, not really-- but the study was financed by the coffee industry. Closer to home, there was a dust-up a few years ago when Elise Bassin, a Harvard doctoral student, had her thesis, which found a statistical link between fluoride and bone cancer, suppressed by her faculty advisor (Chester Douglass), who had an established relationship with several toothpaste manufacturers. Not to foment paranoia, but Big Pharma is not beyond exploiting their control of the knowledge that clinicians use daily to make health care decisions. And you can bet that lobbying groups from both the insurance industry and health care providers are selective in the studies they use to defend their political agendas. (The American Dental Association, whose political leanings place them slightly to the right of Attila the Hun, are just as culpable.)

There have been several attempts to make it mandatory for all clinical trials to be entered in a federal central clinical trials registry. This would require that there be transparency as to the methods used by the studies so that their results can be interpreted in the context of their impartiality and scientific rigor. As you might imagine, corporate-based researchers are strongly resisting this idea.

The bottom line here is: all study results, conclusions, or "facts" should only be considered after a best-effort attempt to assess the quality of the study protocol; their credibility or veracity should never be assumed without this important step. A study should never be taken at face value from a simple abstract, from an unauthoritative source on the internet, or from a sound byte on the evening news. If you are not qualified to assess the rigor of the scientific method employed by the study, defer to the guidance of someone who is qualified (your doctor, your dentist, your science professor, etc.)

And most of all-- view all current conventional wisdom as something that is in a state of flux. Be prepared to discard old cherished "beliefs" when new reliable information suggests that it is appropriate to do so. Just remember that line in Woody Allen's futuristic comedy "Sleeper", in which party guests, passing around recreational drugs, remarked to each other "...remember when they used to think this was bad for you?" "Facts" are a lot like that.