The Sinking Ship of Science: Suggestions for improvement of the peer-review system

I have written a letter to Dr. Eric Eich, editor in chief of Psychological Science. I have proposed a suggestion for improvement of the current peer-review system. First I outlined the problems associated with the current system. Peer-review is underappreciated in the current system: it is often seen as a burden and has to be done in one’s spare time, and is not much rewarded. Also, the author often does not know who the reviewer is. Therefore, it is easy for the reviewer to not spend a lot of time on the review, or even be plain mean. Also, the lack of time spent on the review, makes it more likely for studies in which the researchers have engaged in questionable researchers to slip through.

I have proposed a new peer-review system. In this system, the reviewer does not know who the author is, to prevent certain biases (such as the status of the author, whether he has published many articles in high-impact journals, et cetera). Reviewers receive rewards in the form of review-credits. The authors assigns 1-4 review-credits to the reviewer, based on the quality of the feedback and the usefulness of the review. The editor can then subtract or add one research credit, based on whether the review contains a clear recommendation for the editor concerning publication. The reviewer eventually thus ends up with 0-5 review-credits.

These credits become publicly available, at first on a website but could perhaps also become visible in search engines, next to the number of publications a researcher has. This is a scientist’s review index, with the total number of credits, the number of articles reviewed, and the average number of credits obtained per article. Universities may eventually consider including reviewing in the job descriptions and thus paying for reviews, as good reviewers have a high status.

The review-index will result in reviewers doing a better job on reviews. This results in higher quality of published research articles, as questionable articles are less likely to slip through. Therefore, also authors need to pay attention to clear writing and the prevention of QRPs. Also, the overall review quality will be enhanced, as extrinsic motivation is stimulated. This way, the ship of science will be prevented from sinking.

Data sharing?

Today this article appeared on Retraction Watch, about an odd data sharing policy. The original authors had stated that if the data were to be used by other researchers, the original team should be named as co-authors. Luckily, it appeared to be a mistake in wording and the authors revised the passage. However, might there be any similar policy statements making data sharing more complicated?

What exactly is ‘good science’?

In a recent article in De Volkskrant (September 4, 2014), author Maarten Keulemans states that science is influenced too much by common society. The average citizen does not seem to care about genetically manipulated mice, or the exact role of the release of a certain neurotransmitter in a not yet clearly identified group of cells deep in the human brain. It looks like science is taking this message. For example, Twente University has set up a technology lab to quickly respond to society’s requirements.

In my head, this released a stream of questions. I started wondering what the role of science is. What do we want to know, and why? Do we want to know something just for the sake of knowing it? What is the relevance of the research being done? Should that relevance matter? Should we investigate everything we can think of? Should science be influenced by society at all? And, a question that Jet Bussemaker (current minister of Education, Culture and Science) poses as well, if we can think of something, is it necessary to actually create it? Does society really need the iWatch and yet another version of the iPhone? How involved should (large) companies be in research? Does it make sense to get involved in ‘questionable research practices’, given your goal of knowledge-acquiring? The main question here is about what makes good science, and I’m curious about your opinions. But whatever your opinion of ‘good science’, I suspect that in the end, nobody really profits from untrue knowledge resulting from questionable research practices.