Questionable research practices

I work in the Methodologiewinkel, where psychology students can seek advice from other students on their research methods and data analysis. My colleagues and I often encounter a specific pattern in the analysis strategies among undergraduate (but also graduate!) psychology students. There is a tendency to gather as many variables as possible, without a clear rationale on how they link to the purpose of the study. Consequently, many models are tested during the analysis (after all, one has not took the trouble to gather so many variables for nothing). Moreover, and importantly, many students seem puzzled when they are reminded that they should report every exploratory finding as such. Albeit showing a genuine wish to find something important in the data, students don’t seem to understand what it really means to have to separate exploratory form confirmatory findings. Their behaviour cannot be considered cheating, but surely questionable.

For my final paper, I evaluated 14 statistics and methodology books to see whether they address 4 of such questionable research practices (those used in Simmons, Nelson, & Simonsohn ,2011; Testing on two or more dependent variables, testing additional subjects or optional stopping, including covariates ad hoc, dropping conditions/ not reporting them) and how elaborate their chapters are on the ethical implications and the ‘do’s and don’ts’  in research. Surprisingly, I found some clearly wrong, some at least misleading, and only a few good accounts that address the issues in depth, giving illustrative examples and useful practical solutions. Although all books do, in fact, discuss exploratory versus confirmatory research, the explanations remain abstract, and without concrete practical implications.

In one of the books I stumbled over a discussion on the possible reasons of fraud, where the ‘publish or perish’ – culture  is seen as one of the contributing factors. This is the first time that I have never, during my bachelor in Vienna or the master in Amsterdam, come across a note about what kind of pressures you might face in your later research career. Never are we really told about those daily issues that De Vries, Anderson & Martinson (2006) have called the “normal misbehaviours”:  Conflicts of interest among colleagues, ‘rules of conduct in a lab’, the normal practice of deciding upon authorship, possibilities on how to behave if someone is being cut out,  how to keep proper research records….etc. While there might be many abstract discussions about the ethics of research, there are no practical guidelines that would prepare young researchers to deal with those kinds of social conflicts (and how to avoid that they influence the quality of research). Maybe we cannot avoid such ‘misbehaviours’ entirely, as De Vries et al. (2006) argue.  But we can cat least raise awareness for them among the new generations of students

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366

De Vries, R., Anderson, M.S.& Martinson, B.C.(2006).Normal misbehavior: Scientists talk about the ethics of research.  Journal of Empirical Research on Human Research Ethics: JERHRE,1, 43 – 50



Replication of Roskes et al. (2011)


In yesterday’s lecture, the different groups presented their work of the past weeks, some preliminary results, and an evaluation of the process of our replication study. The results will be more conclusive at the end of the week, but much can already be said about the process. I think we all agree that it was not easy: Working within a group of 5 already takes a lot of effort, but coordinating 15 people seems to be an order of magnitude harder! Here it becomes more obvious how important it is, for example, to keep proper analysis and work protocols, or how to write clear and informative emails, so that work can be smoothly taken over by a different group (things I did not find in any books on research methodology I scanned for my final paper). Nevertheless, we can be proud that we have managed to dive into the literature and set up the theoretical framework,  gather 500 penalty shots, let 7 people rate them and do the first sets of analyses in about 5 weeks’ time. And the possibility to make a contribution to the PsychFileDrawer project is a rewarding prospect. Finally, we saw how it feels like to carry out a study sticking exactly to the preregistered report. During my onderzoekspracticum in my bachelor, I remember being astonished by the amount of fellow students delivering sloppy work,  often with the excuse that ‘it is just for the practicum, nobody will ever read it anyway.’ I think involving students in a study that will be posted online beforehand could be a way of making students feel more responsibility of doing research properly. I think we did a good job overall, and I can imagine groups of students everywhere carrying out some of the many awaiting exact replications in psychology and posting them online.

Roskes, M., Sligte, D., Shalvi, S., & De Dreu, C. K. (2011). The right side? Under time pressure, approach motivation leads to right-oriented bias. Psychological science, 22, 1403-1407.