Blog on replication; Discussion with EJ Wagenmakers & Michelle Nuijten

Hi everybody.

In this blog we review the major themes in the discussion of Thursday 25/9/2014.

Why (not) replicate?

  • A true finding should be replicable.
  • Replication is no QRP’s detector: if you want to you can make a finding happen. However, without QRP’s (and fraud) replication is meaningful.

How can we stimulate replication:

  • Reward system: give extra point to researchers that replicate findings.
  • Punish system: punish studies that cannot be replicated. For example, by linking the original study to the failed replication (there was no consensus about this point).
  • Change the research policies: replicate your study a couple of times before publication to be sure that you publish a meaningful finding.
  • Let students do a replication in their internship instead of (mostly failing) new research.

It is impossible to replicate all studies. Editors could determine which study needs to be replicated and researchers need to focus their replications on:

  • Studies on which policies are based.
  • High impact studies.
  • Implausible findings.

Lastly we discussed some changes that would benefit science and psychology:

  • Researchers need to change their mindsets: they do research in isolation. We need building blocks that combine expertise and knowledge.
  • Theory can function as a boundary in maintaining quality within science. However, there are too many useless theories: every finding can be explained by a different theory. We need better theories, for example theory based on mathematical models.
  • Supervisors and teachers need training in doing research the right way: preventing QRP’s.
  • Every researcher need to use Bayesian statistics (of course…).
  • Studies need to be pre-registered.
  • Prevent testing small samples because they will always bias the results (unless you pre register).
  • Every department needs a methodologist. UvA is planning to set up a ‘methodenleer winkel’ for the staff to get methodological advice.
  • Attitude change: we won’t discover large effects anymore, researchers will from now on probably discover only small effects.
  • Combine research effort: results are no personal baby.

Finally an optimistic note: We are moving in the right direction, in the sense of creating a lot of awareness and good initiatives to improve replication and replicability in psychology.

Regards,

Sarah, Noor, Lukas, Bianca & Bob

Food for more thought

Hi all,

This paper might be of interest to you:

Ethical Issues and Guidelines for Conducting Data Analysis in Psychological Research, Rachel Wasserman (2013).

One of her questions is: How can researchers know when a questionable method they use to perform data analyses reflects a difference of opinion or a possible violation of ethics? 

Maybe this links works: EthicalIssues-Wasserman

Happy weekend,

Eline