Educating the Educators

In this essay, I argue that the scientific world would benefit from a better research education for both students and researchers. Within the University of Amsterdam, re-introducing the so called ‘Research Practical’ is a very practical and easy-to-implement way to acquaint students with several problems in science, such as lack of replication. Also, in this course students can learn how to deal with things like non-replication and finding of a null effect.

The university should also take care to educate their researchers. This can be done through courses in methodology and statistics, or through an open forum on which researchers can discuss their methodological and statistical considerations with their colleagues. Educating researchers does not only benefit science: since mentors have a great influence on their pupils, good researchers will make more good researchers. And if there is one thing that science needs, it is good researchers.

Pre-registration: improvement or imprisonment?

As I dare to assume that most readers of this article are advocates of pre-registration, I will not elaborate on how amazing it is. On the other hand, I thought I’d try to be critical for once and see whether there was something to be said against pre-registration. And there was.

On the website of the Times Higher Education, a section of the Times which is about higher education, Sophie Scott, the deputy director of the Institute of Cognitive Neuroscience at University College London, says that pre-registration would put science ‘in chains’ (Scott, 2013). She does have several arguments for this position, however, they did not convince me. To spare you the effort of having to read the article yourself I’ll state them here and, for your convenience, will also explain why I think they are not too strong.

At the start, she reacts to an open letter in the Guardian, which advocates a specific form of study pre-registration, called registered reports. With registered reports, the research proposals are reviewed before data collection. If the researchers think the hypotheses and method are sound, the article is accepted ‘in-principle’. This means that publication of the article is virtually guaranteed, no matter what results are found (Chambers et al., 2013). Miss Scott has several problems with this. Firstly, she states that it would make papers more one-dimensional in perspective, because it would limit the ‘more speculative aspects of data interpretation’. However, in registered reports, nothing is said in advance about how the data should be interpreted. All that is examined is whether the theory and method seem sound, and researchers can still make speculative interpretations of their data. The only difference with non-registered reports is that it is clear to everyone that these interpretations are speculative and were not hypothesized at the outset of the study. Secondly, because registered reports do form some kind of contract between the journal and the researcher, it would constrain the researcher’s freedom to choose the journal that best fits their results. However, I think this is a small price to pay for increased scientific integrity. Also, it is inevitable that researchers have certain expectations when they are designing a study, and they are still free to choose the journal that best fits their expectations of what is going to happen.

Furthermore, Miss Scott is afraid that reviewers would, more than ever, rely heavily on the reputation of the scientists who submitted the proposal, and this would give young researchers no chance to get published. This would be because, without results, the reviewers would have ‘nothing to go on’… which is, of course, nonsense. This statement shows a misunderstanding of the entire problem for which pre-registration is supposed to be a solution and paints the image of what reviewers value in reports: results. Good scientific research should be at least as much about the method as it should be about the results. Without a good method, your results are worthless, but with a sound method, your results are valuable, even if they show no effect. The fact that Miss Scott comes up with this argument shows that there should be a huge attitude-change in the scientific community, where method is esteemed at least as highly as results. And because one can expect reviewers to have received at least a basic education on method and statistics, they will have a lot to go on if they are presented with just that. Then, it is stated that this system would give unsympathetic reviewers the chance to veto studies at the outset. However, in the current system, unsympathetic reviewers can keep a study from being published. I do not really see whether the latter one should be preferred: from an economic point of view the former would be even better, because it would keep a lot of money from being spent on an unpublished study.

Also, it is stated that in field like cognitive neuroscience and psychology (which happen to be Miss Scott’s fields of expertise) the pre-registration model would not work because many studies are not designed to test hypotheses but are, for example, observational. However, I feel that this does not form an obstacle. Observational studies have a method: if one is, for example, observing whether there is a difference in the extraversion of children from two different schools, one has to at least code some behaviours that one would count as evidence of extraversion (which are likely to be taken from theory) and later statistically analyse these to see whether there are differences. To me this seems like the start a lovely proposal to be submitted to one’s favourite journal on child psychology.

Miss Scott’s proposition that pre-registration puts science in chains is true in some way: the freedom of scientists while doing their research is severely limited by obliging them to stick to their proposals. However, scientists have in fact always had that obligation, but until pre-registration came along there never was a method to check if they were actually doing this. Apparently, some people can not handle the responsibility to do sound scientific research without feeling the hot breath of the rest of the scientific community in their necks and for the ones who can, the pre-registration model will cause them to make minimal changes to their way of doing research.

In summary, I think that it is good to not just accept anything that is shouted by researchers. However, Miss Scott did not manage to convince me. How about you?

References

Chambers, C. et al. (June 5th, 2013). Trust in science would be improved by study pre-registration. Retrieved from http://www.theguardian.com/science/blog/2013/jun/05/trust-in-science-study-pre-registration

Scott, S. (2013). (July 25th, 2013). Pre-registration would put science in chains. Retrieved from http://www.timeshighereducation.co.uk/comment/opinion/science-in-chains/2005954.article