Ten years ago the “New Scientist” magazine ran an article with the headline “Most scientific papers are probably wrong” (see link). The article referred to a paper by John Ioannidis (which is available here). In his summary of the paper, Mr. Ioannidis writes:
“There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias”.
The New Scientist article includes the paragraph:
Most published scientific research papers are wrong, according to a new analysis. Assuming that the new paper is itself correct, problems with experimental and statistical methods mean that there is less than a 50% chance that the results of any randomly chosen scientific paper are true.
Mr. Ioannidis is an epidemiologist, but his comments could just as well apply to engineering research. In the field of highway engineering, the Canadian author Ezra Hauer has been writing about his doubts with research since well over 30 years. The 2014 version of his paper on “Fishing for Safety Information in the Murky Waters of Research Reports” is a readable and interesting contribution to the general topic.
One risk with doubtful research – when published in respectable journals – is that it may be accepted at face value. Others may build on it, and its findings may crop up eventually in techbooks and design standards. In this case, perhaps engineers should maintain a healthy suspicion even of recommendations in highway design standards. Who knows where the responsibility for road accidents lies – with the road user, with the road designer, or with the committees that develop design standards?