Doubtful research

Ten years ago the “New Scientist” magazine ran an article with the headline “Most scientific papers are probably wrong” (see link). The article referred to a paper by John Ioannidis (which is available here). In his summary of the paper, Mr. Ioannidis writes:

 “There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias”.

The New Scientist article includes the paragraph:

Most published scientific research papers are wrong, according to a new analysis. Assuming that the new paper is itself correct, problems with experimental and statistical methods mean that there is less than a 50% chance that the results of any randomly chosen scientific paper are true.

Mr. Ioannidis is an epidemiologist, but his comments could just as well apply to engineering research. In the field of highway engineering, the Canadian author Ezra Hauer has been writing about his doubts with research since well over 30 years. The 2014 version of his paper on “Fishing for Safety Information in the Murky Waters of Research Reports” is a readable and interesting contribution to the general topic.

Comment

One risk with doubtful research – when published in respectable journals – is that it may be accepted at face value. Others may build on it, and its findings may crop up eventually in techbooks and design standards. In this case, perhaps engineers should maintain a healthy suspicion even of recommendations in highway design standards.   Who knows where the responsibility for road accidents lies – with the road user, with the road designer, or with the committees that develop design standards?

Advertisements

About roadnotes

Robert Bartlett is an international consultant with over 30 years of professional experience as a highway and traffic engineer with leading companies and organisations in several countries, including Germany, China (Hong Kong), Qatar and the UK. Specialised in urban studies, transport and the use of GIS, research has included new ideas on subjects such as the study of social justice using GIS, the dimensions of vehicles, and comparative geometrics (highways and transport).
This entry was posted in highway design standards and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s