I work in the Methodologiewinkel, where psychology students can seek advice from other students on their research methods and data analysis. My colleagues and I often encounter a specific pattern in the analysis strategies among undergraduate (but also graduate!) psychology students. There is a tendency to gather as many variables as possible, without a clear rationale on how they link to the purpose of the study. Consequently, many models are tested during the analysis (after all, one has not took the trouble to gather so many variables for nothing). Moreover, and importantly, many students seem puzzled when they are reminded that they should report every exploratory finding as such. Albeit showing a genuine wish to find something important in the data, students don’t seem to understand what it really means to have to separate exploratory form confirmatory findings. Their behaviour cannot be considered cheating, but surely questionable.
For my final paper, I evaluated 14 statistics and methodology books to see whether they address 4 of such questionable research practices (those used in Simmons, Nelson, & Simonsohn ,2011; Testing on two or more dependent variables, testing additional subjects or optional stopping, including covariates ad hoc, dropping conditions/ not reporting them) and how elaborate their chapters are on the ethical implications and the ‘do’s and don’ts’ in research. Surprisingly, I found some clearly wrong, some at least misleading, and only a few good accounts that address the issues in depth, giving illustrative examples and useful practical solutions. Although all books do, in fact, discuss exploratory versus confirmatory research, the explanations remain abstract, and without concrete practical implications.
In one of the books I stumbled over a discussion on the possible reasons of fraud, where the ‘publish or perish’ – culture is seen as one of the contributing factors. This is the first time that I have never, during my bachelor in Vienna or the master in Amsterdam, come across a note about what kind of pressures you might face in your later research career. Never are we really told about those daily issues that De Vries, Anderson & Martinson (2006) have called the “normal misbehaviours”: Conflicts of interest among colleagues, ‘rules of conduct in a lab’, the normal practice of deciding upon authorship, possibilities on how to behave if someone is being cut out, how to keep proper research records….etc. While there might be many abstract discussions about the ethics of research, there are no practical guidelines that would prepare young researchers to deal with those kinds of social conflicts (and how to avoid that they influence the quality of research). Maybe we cannot avoid such ‘misbehaviours’ entirely, as De Vries et al. (2006) argue. But we can cat least raise awareness for them among the new generations of students
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366
De Vries, R., Anderson, M.S.& Martinson, B.C.(2006).Normal misbehavior: Scientists talk about the ethics of research. Journal of Empirical Research on Human Research Ethics: JERHRE,1, 43 – 50