Current publication practices may distort science

Image via Wikipedia

From the excellent Open Access News, I chanced across the article Why Current Publication Practices May Distort Science by Neal S. Young, John P. A. Ioannidis, Omar Al-Ubaydli. I am already familiar with many of the problems that plague the existing scholarly publishing system, and have been very interested in approaches such as open notebook science which might possibly address some of these faults. This current article uses an economic framework drawn from auctions to show that articles published in the top journals are biased towards outlier findings, and mention that many of the top empirical articles published during a certain time period, were later shown to have much weaker correlations. The result is that relying on the positive findings published in top journals, and not having access to all the negative findings, which have a hard time getting published, results in poor science, or even worse in the case of medicine.

The authors also mention how the system is built on the artificial limitations of the paper age, and how journals are proud of their high rejection rates (similar to how universities in the US are proud of how many students they reject). They suggest that instead, a peer review system’s task should be to see if a journal article is sound empirically, and if it is, then it should be published - at least online. This would be the equivalent of a pass/fail system in say vocational training in Norway: either you are qualified as a carpenter, or you are not. We don’t judge whether you are the best carpenter this year.

Of course a large part of the problem - which the authors don’t really mention - is that scholarly publishing plays such a huge role in tenure reviews and promotions for academics (and in many countries, also in determining government funding). Thus, similarly to education, the journal system is given a task that it was not really designed for - not merely to sort out the chaff from the wheat, but also to prioritize the juiciest (?) wheat into the best journals, so that those professors can get higher compensation. This is similar to the sorting function of education for students.

All in all, a very well written paper, short and concise, with some interesting new approaches, that is well worth a read.

Stian

This entry was posted on Tuesday, October 7th, 2008 at 16:22 and is filed under education/academia, open access. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.