A Healthy Dose of Skepticism

The Wall Street Journal recently published two pieces (one article, and one on the WSJ Health Blog) on the basic theme of whether scientific research can be trusted, and referring to an essay by John P. A. Ioannidis, Why Most Published Research Findings Are False.

Ioannidis's points are neatly summed up by this passage in the introduction to the freely available essay:

The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance.

In other words, the methods matter, as does bias and the type and size of the finding. In his recent article, in the Wall Street Journal, Robert Lee Hotz points out that while research methods matter, it is also procedurally difficult for peer reviewers and editors to tease out inappropriate techniques and conclusions:

"...findings too rarely are checked by others or independently replicated. Retractions, while more common, are still relatively infrequent. Findings that have been refuted can linger in the scientific literature for years to be cited unwittingly by other researchers, compounding the errors...No one actually knows how many incorrect research reports remain unchallenged."

In addition, trials that do not demonstrate findings that are suitable to the trial sponsors may be shuttered and never reported in the literature. A related and lengthy piece in a recent New York Times Magazine, "Do We Really Know What Makes Us Healthy?" used hormone replacement therapy as an example throughout to illustrate why it can be difficult to make sense of research findings, including that the first, dramatic reports are likely to garner media attention, while later clarifications may not.

The bottom line? A healthy dose of skepticism is needed when reading research reports, and a good working knowledge of bias, research methodology, and other methods can go a long way.