Webdiary - Independent, Ethical, Accountable and Transparent
header_02 home about login header_06
sidebar-top content-top

To Err is Science

John P.A. IoannidisJohn P.A. Ioannidis is Professor and Chairman at the Department of Hygiene and Epidemiology at the University of Ioannina School of Medicine in Greece and adjunct Professor of Medicine at Tufts University School of Medicine in Boston, USA.

by John P.A. Ioannidis

Until now, discovery was often considered the main goal of medical science. But nowadays discovery is almost too easy. Anyone with a little funding and a few biological specimens in a refrigerator can make thousands of postulated "discoveries."

Indeed, the number of research questions that we can pose is increasing exponentially. Medical kits the size of a thumbnail can measure a million different biological factors on an individual with an infinitesimal amount of blood. A million research questions can be asked on the spot. But even with proper statistical testing, many tens of thousands of these biological factors may seem to be important due to mere chance. Only a handful of them really will be. The vast majority of these initial research claims would yield only spurious findings.

So the main issue nowadays is to validate "discoveries" by replicating them under different settings. Several different teams of researchers need to see them "work" again and again using common rules. Moreover, all the teams should agree not to select and report only the data that seem most impressive. With selective reporting, we would end up with a long list of all the false discoveries made across all research teams, with only a few true findings buried among this pile of non-replicated waste.

In fact, empirical data suggest the significance of this danger. In a paper in the Journal of the American Medical Association published in July 2005, I showed that refutation is very common, even for the most prestigious research findings. I examined the 45 clinical research findings that had received the greatest recognition in the scientific world, as documented by the number of times other scientists had cited them over the last 15 years.

Even with the most robust types of research – for example, randomized clinical trials – one of four of these results had already been found to be wrong or potentially exaggerated within a few years after publication. For epidemiology (e.g. studies on vitamins, diet, or hormones in terms of their association with health outcomes in the general population), four-fifths of the most prestigious findings were rapidly contradicted. For molecular research, in the absence of extensive replication, the refutation rate may occasionally exceed 99%.

But we should not panic. It is to be expected that the majority of research findings are rapidly contradicted and refuted; indeed, it is part of how progress of science occurs. However, we need to adapt to this situation. Instead of taking scientific evidence as dogma, we should consider it as tentative information that should be ascribed a level of credibility.

There is nothing wrong with disseminating scientific information that has a credibility of 10%, or even 1%. Sometimes, this will be the best evidence we have. But we should get used to understanding that some research findings have very low credibility, while others may be more likely to stand the test of time. Scientists themselves may be able to ascribe these levels of credibility to their own work in fairness, if they describe in detail what they set out to do, and how they did it.

Science is a noble pursuit, but genuine progress in scientific research is not easy to achieve. It requires a lot of time, continuous effort, uncompromising integrity, appropriate funding and material support, and unwavering commitment. Proposed scientific advances require careful validation and replication by independent scientists. Scientific knowledge is never final, but it evolves continuously. This is part of the great fascination of science, and it fosters liberty of thought.

While these principles are probably well known to serious scientists, they are often forgotten when scientific information is disseminated. Our society is awash with inflated information, which is inherent to efforts in many human activities – entertainment, law courts, stock markets, politics, and sports, to name but a few – to gain greater public attention in the framework of mass civilization.

But it would be a damaging to expect science to "show off" in this way. Exaggeration contradicts the key hallmarks of scientific reasoning: critical thinking and careful appraisal of the evidence.

Copyright: Project Syndicate, 2006.

[ category: ]
© 2005-2011, Webdiary Pty Ltd
Disclaimer: This site is home to many debates, and the views expressed on this site are not necessarily those of the site editors.
Contributors submit comments on their own responsibility: if you believe that a comment is incorrect or offensive in any way,
please submit a comment to that effect and we will make corrections or deletions as necessary.
Margo Kingston Photo © Elaine Campaner

Recent Comments

David Roffey: {whimper} in Not with a bang ... 13 weeks 6 days ago
Jenny Hume: So long mate in Not with a bang ... 14 weeks 14 hours ago
Fiona Reynolds: Reds (under beds?) in Not with a bang ... 14 weeks 2 days ago
Justin Obodie: Why not, with a bang? in Not with a bang ... 14 weeks 2 days ago
Fiona Reynolds: Dear Albatross in Not with a bang ... 14 weeks 2 days ago
Michael Talbot-Wilson: Good luck in Not with a bang ... 14 weeks 2 days ago
Fiona Reynolds: Goodnight and good luck in Not with a bang ... 14 weeks 3 days ago
Margo Kingston: bye, babe in Not with a bang ... 15 weeks 15 hours ago