Until now, discovery was often considered the main goal of medical science. But nowadays discovery is almost too easy. Anyone with a little funding and a few biological specimens in a refrigerator can make thousands of postulated “discoveries.”
Indeed, the number of research questions that we can pose is increasing exponentially. Medical kits the size of a thumbnail can measure a million different biological factors on an individual with an infinitesimal amount of blood. A million research questions can be asked on the spot. But even with proper statistical testing, many tens of thousands of these biological factors may seem to be important due to mere chance. Only a handful of them really will be. The vast majority of these initial research claims would yield only spurious findings.
So the main issue nowadays is to validate “discoveries” by replicating them under different settings. Several different teams of researchers need to see them “work” again and again using common rules. Moreover, all the teams should agree not to select and report only the data that seem most impressive. With selective reporting, we would end up with a long list of all the false discoveries made across all research teams, with only a few true findings buried among this pile of non-replicated waste.
In fact, empirical data suggest the significance of this danger. In a paper in the Journal of the American Medical Association published in July 2005, I showed that refutation is very common, even for the most prestigious research findings. I examined the 45 clinical research findings that had received the greatest recognition in the scientific world, as documented by the number of times other scientists had cited them over the last 15 years.
Even with the most robust types of research – for example, randomized clinical trials – one of four of these results had already been found to be wrong or potentially exaggerated within a few years after publication. For epidemiology (e.g. studies on vitamins, diet, or hormones in terms of their association with health outcomes in the general population), four-fifths of the most prestigious findings were rapidly contradicted. For molecular research, in the absence of extensive replication, the refutation rate may occasionally exceed 99%.
But we should not panic. It is to be expected that the majority of research findings are rapidly contradicted and refuted; indeed, it is part of how progress of science occurs. However, we need to adapt to this situation. Instead of taking scientific evidence as dogma, we should consider it as tentative information that should be ascribed a level of credibility.
At a time when democracy is under threat, there is an urgent need for incisive, informed analysis of the issues and questions driving the news – just what PS has always provided. Subscribe now and save $50 on a new subscription.
Subscribe Now
There is nothing wrong with disseminating scientific information that has a credibility of 10%, or even 1%. Sometimes, this will be the best evidence we have. But we should get used to understanding that some research findings have very low credibility, while others may be more likely to stand the test of time. Scientists themselves may be able to ascribe these levels of credibility to their own work in fairness, if they describe in detail what they set out to do, and how they did it.
Science is a noble pursuit, but genuine progress in scientific research is not easy to achieve. It requires a lot of time, continuous effort, uncompromising integrity, appropriate funding and material support, and unwavering commitment. Proposed scientific advances require careful validation and replication by independent scientists. Scientific knowledge is never final, but it evolves continuously. This is part of the great fascination of science, and it fosters liberty of thought.
While these principles are probably well known to serious scientists, they are often forgotten when scientific information is disseminated. Our society is awash with inflated information, which is inherent to efforts in many human activities – entertainment, law courts, stock markets, politics, and sports, to name but a few – to gain greater public attention in the framework of mass civilization.
But it would be a damaging to expect science to “show off” in this way. Exaggeration contradicts the key hallmarks of scientific reasoning: critical thinking and careful appraisal of the evidence.
To have unlimited access to our content including in-depth commentaries, book reviews, exclusive interviews, PS OnPoint and PS The Big Picture, please subscribe
In 2024, global geopolitics and national politics have undergone considerable upheaval, and the world economy has both significant weaknesses, including Europe and China, and notable bright spots, especially the US. In the coming year, the range of possible outcomes will broaden further.
offers his predictions for the new year while acknowledging that the range of possible outcomes is widening.
Until now, discovery was often considered the main goal of medical science. But nowadays discovery is almost too easy. Anyone with a little funding and a few biological specimens in a refrigerator can make thousands of postulated “discoveries.”
Indeed, the number of research questions that we can pose is increasing exponentially. Medical kits the size of a thumbnail can measure a million different biological factors on an individual with an infinitesimal amount of blood. A million research questions can be asked on the spot. But even with proper statistical testing, many tens of thousands of these biological factors may seem to be important due to mere chance. Only a handful of them really will be. The vast majority of these initial research claims would yield only spurious findings.
So the main issue nowadays is to validate “discoveries” by replicating them under different settings. Several different teams of researchers need to see them “work” again and again using common rules. Moreover, all the teams should agree not to select and report only the data that seem most impressive. With selective reporting, we would end up with a long list of all the false discoveries made across all research teams, with only a few true findings buried among this pile of non-replicated waste.
In fact, empirical data suggest the significance of this danger. In a paper in the Journal of the American Medical Association published in July 2005, I showed that refutation is very common, even for the most prestigious research findings. I examined the 45 clinical research findings that had received the greatest recognition in the scientific world, as documented by the number of times other scientists had cited them over the last 15 years.
Even with the most robust types of research – for example, randomized clinical trials – one of four of these results had already been found to be wrong or potentially exaggerated within a few years after publication. For epidemiology (e.g. studies on vitamins, diet, or hormones in terms of their association with health outcomes in the general population), four-fifths of the most prestigious findings were rapidly contradicted. For molecular research, in the absence of extensive replication, the refutation rate may occasionally exceed 99%.
But we should not panic. It is to be expected that the majority of research findings are rapidly contradicted and refuted; indeed, it is part of how progress of science occurs. However, we need to adapt to this situation. Instead of taking scientific evidence as dogma, we should consider it as tentative information that should be ascribed a level of credibility.
HOLIDAY SALE: PS for less than $0.7 per week
At a time when democracy is under threat, there is an urgent need for incisive, informed analysis of the issues and questions driving the news – just what PS has always provided. Subscribe now and save $50 on a new subscription.
Subscribe Now
There is nothing wrong with disseminating scientific information that has a credibility of 10%, or even 1%. Sometimes, this will be the best evidence we have. But we should get used to understanding that some research findings have very low credibility, while others may be more likely to stand the test of time. Scientists themselves may be able to ascribe these levels of credibility to their own work in fairness, if they describe in detail what they set out to do, and how they did it.
Science is a noble pursuit, but genuine progress in scientific research is not easy to achieve. It requires a lot of time, continuous effort, uncompromising integrity, appropriate funding and material support, and unwavering commitment. Proposed scientific advances require careful validation and replication by independent scientists. Scientific knowledge is never final, but it evolves continuously. This is part of the great fascination of science, and it fosters liberty of thought.
While these principles are probably well known to serious scientists, they are often forgotten when scientific information is disseminated. Our society is awash with inflated information, which is inherent to efforts in many human activities – entertainment, law courts, stock markets, politics, and sports, to name but a few – to gain greater public attention in the framework of mass civilization.
But it would be a damaging to expect science to “show off” in this way. Exaggeration contradicts the key hallmarks of scientific reasoning: critical thinking and careful appraisal of the evidence.