More reasons for doing nothing: untrustworthy studies

“Much of what medical researchers conclude in their studies is misleading, exaggerated, or flat-out wrong” says an article in the November, 2010 issue of The Atlantic. The article focuses on John Ionnidis, who is, among other things, the director of the Stanford Prevention Research Center at Stanford University School of Medicine. As a “meta-researcher,” he and his team apply rigorous statistical analysis to try to verify the evidence reported in medical journals.

After poring over journals to determine the credibility of medical research, he and his team found that as much as 90 percent of the information that doctors rely on is flawed. (Other meta-researchers are also finding disturbing high rates of error in medical literature.) “At every step in the process, there is room to distort results, a way to make a stronger claim or to select what is going to be concluded,” notes Ionnidis. For example, researchers can skew results by the ways in which they pose questions, recruit patients, select factors to measure, analyze the data, and present results.

The reason researchers distort results is primarily to increase the paper’s chance of getting published and to get themselves funded. (To stay afloat, researchers need to get their work published in well-regarded journals where rejection rates are high.) The papers that get published are those with eye-catching findings. As it turns out, when studied rigorously, the great majority of eye-catching findings collapse under the weight of contradictory data. And, of course, drug studies have added the corruptive force of financial conflict of interest by manipulating data to make their drugs look good.

As one of the doctors on Ionnidis’ team says, “When you look the papers up, you often find the drugs didn’t even work better than a placebo. And no one tested how they worked in combination with the other drugs. Just taking the patient off everything can improve their health right away.” 

Ionnidis found that even when a research error has been made known, the original results can persist for years or even decades. In looking at three prominent studies that had been soundly refuted, Ionnidis discovered that researchers continued to cite the original results as correct—in one case 12 years after the results were discredited. (Incidentally, here’s something I learned that was news to me: vitamin E—fish oil—does not help prevent cardiovascular disease.)

Doctors have been trained to order tests and prescribe whatever drugs they believe will affect an out-of-whack test number. They're not trained to study the research papers that helped make drugs they prescribe the standard of care—a time-consuming task. So the drug habits persist, helped by the fact that patients often don’t like it when they’re taken off their drugs. They find their prescriptions reassuring.

I highly recommend the article.

Next week: over-testing run amok--an example.

For an introduction to this blog, see I Just Say No; for a list of blog topics, click the Topics tab.