The Case for Theoretical Psychology

Kurt Lewin, one of the founders of social psychology, once stated that “there is nothing as practical as a good theory”. Yet today, many decades later, psychology appears to have all but forgotten about this advice: there is not much theory, and what little there is, is rarely good.

In fact, psychology is a profoundly experimentalist discipline: even those who do appreciate theory appear to do so mostly because it can inform future experimental research.

In contrast, I believe that abstraction is itself a fundamental goal of science, and as such psychologists should aim to construct theories that explain as much of the world as possible with as little as possible: broad in appeal, parsimonious in execution.

To achieve this, theories need to be both abstract (i.e., removed from the operational level) and formal (i.e., stated in mathematical, logical, or comparable terms). Today, few psychologists have the incentives, willingness, or ability to create such theories.

In fact, current structures incentivise the construction of overly narrow theories which can then be molken for experimental publications. Terror management theory, for example, describes a tiny sliver of the human experience and only one of many meaning maintance mechanisms; but there would have been little incentive for its founders to more on to a more abstract theory.

This leads to ideological-theoretical fragmentation of psychology, in which each theory has its adherents, mostly clustered around the institution of its founders, but many theories are outdated or redundant – and nobody is cleaning up.

To counteract this situation, I believe we need theoretical psychology as an independent subdiscipline. Instead of having experimenters dabble as occasional theorists, theory construction should be a specialised skill to be learned early on, then pursued as a career.

Creating a new subdiscipline is not an undertaking any one actor should engage in. Rather, I believe universities, through introducing explicit training in theory construction, funding agencies, through supporting research and outreach projects aimed at creating best practices for theory construction as well as work towards ‘cleaning out’, unifying, and integrating theories, and publishers and editors, through demanding theoretical publications to be formalised and making space for formal theorising, can all support the development of better theory construction in psychology.

Letter of Concern

Hi all,

I decided to write my letter of concern to Agnetha Fischer. As an example of what I think psychological research can be I used the Pikkety debate. A debate that is characterized by a high technical and social-political discourse but with transparency, of both data and method, as a driving force behind it. I describe what I have been sharing with you in the poster presentation on high competition in psychological science as a possible explanation of scientific misconduct. I mentioned the paradox of the general rejection of the usage of QRP’s together with the estimated high amount of QRP’s. Next to this I used the example of the impact of mentors (both bad and good) to show that self-regulation is apparently too much to expect from researchers themselves and therefore clear rules and arbitration is necessary to minimize misconduct. Because I hold the UvA responsible as the organisation that creates the context of desirable and undesirable behavior of it’s employees I recommend the following:

1. Get informed: Communicate with your employees about the problems psychological science is facing and involve them in policy making on diminishing chances on misconduct like QRP’s. Minimize the chance of feelings of frustration when introducing new strategies.

2. Become an Open Access University: Every UvA-researchers that has published an article should also provide the raw data and data-analyses on the institutes website. This should be an open and interactive environment for registered individuals including those outside the UvA-community: The UvA Open Access website.

3. Scientific Arbitration Team (SA-Team): There should be frequent investigations on published articles on QRP’s and data handling of UvA- employees. This can be done at random just as tax-officers do their audits. This investigations should be lead by the SA-team who are supported by master students of the methodology – and research department. Investigating QRP’s should become a part of the students curriculum because it teaches the students judgement on research and because it will generate a thinking process that will make them reflect on their own integrity.

I included other responsibilities and prospects for future activities for the SA-team which I am happy to share with you while enjoying a drink on a Thursday afternoon.

Most important for me to communicate was that these recommendations should facilitate a transition in which pre-registration and sharing knowledge is the new norm in order to make psychological research more reliable and meaningful. Self-regulation of the individual researchers and the scientific community trough transparency is the key aspect of these “solutions”.  I further discuss expected positive consequences (regulation by self and others, enhancement of UvA’s public profile etc.) and expected negative effects (resistent to change, practical, financial) of these recommendations.

My conclusion of it all:

Although the field of psychology is going true rough times, the UvA could become a precursor in turning the tide. In order to blossom, psychology and the UvA need rigorous change that goes beyond the comfort-zone of the average scientist or organisation. By implementation of new organizational strategies one will enforce a more reliable practice and optimize the conditions for outstanding scientific conduct. I firmly believe that these recommendations will lead to the integrity psychological research needs. That finding the truth becomes once again the core priority of the research practice.”

Cheers and see you around!


Saving Psychology: Changing Research by Changing the Researcher

Between frightening reports of fraud and other scientific misconducts, psychology risks losing face as a science. A main concern is that the hypotheses and explanations provided by psychologists are worthless, as they were obtained through questionable if not outright wrong methods. To remedy this, innovate projects such as the Open Science Framework and Preregistration are now emerging, aimed to improve the system; to impose rules and regulations that create scientific environment wherein researchers’ practices can be monitored more closely, so that dodgy research behaviors like fraud and QRP’s cannot exist, or at least can be detected more easily. One might wonder though, if this will be enough. After all, it was not the system but the researcher that chose to employ faulty research methods, and it may be optimistic to think that the current state of affairs can be improved without changing the researcher as well. For one, having the system make it harder for some faulty research practices to occur may just lead to an increase in others, that can slip past the system (for example, making up data). With this in mind, this paper will focus on improving psychology as a science by educating its researchers. The goal of this education will be to: 1) provide them with the knowledge to make responsible, well-informed research decisions, and 2) stimulate attitudes that will intrinsically motivate researchers to make responsible, well-informed research decisions. This is expected to reduce the use of questionable research practices and fraudulent behaviors in psychology. This blog will go into explaining: why researchers need to change, what aspects need to change and in what manner, and end in suggestions on how to realize these changes.

Why do researchers need to change? As was said before, changing the system may not be enough. Certain faulty research practices, such as altering or making up data (aka fraud) can still occur. Besides which, inhibiting some malpractices may simply lead to an increase in others; after all, researchers who were motivated to obtain certain results before system changes, will still be motivated to do so after. Making it harder to do wrong will not always lead to less wrongdoing. As long as external motivators (for example, publication pressure and publication bias) are still pushing researchers to obtain certain results, researchers will strive for these results, by any means necessary. Educating researchers may however provide researchers with the intrinsic motivation and means to fight these extrinsic motivators; to prioritize the quality of research over personal gain.

What needs to change then, and in what way? As was said before, the goal of this advice plan is to improve the researcher’s knowledge concerning research practices, and change their attitude towards research. The plan is to provide researchers with knowledge concerning research practices, both accepted and faulty research practices, in terms of research methods, designs, analyses and inferences. The focus will not only be bad research practices, and why they should not be used, but also good research practices, and when they can and cannot be used, why, and how to use them. For example, one can address that deleting data points that don’t support one’s hypothesis is considered a questionable research practice, and why, and what is the proper way to handle if the data does not support the research hypothesis. This will provide researchers with sufficient knowledge to make well-informed, responsible decisions when involved in research. Furthermore, the plan is to change researcher’s attitude towards research, or more specifically, to: stimulate a critical attitude, a willing to face one’s own shortcomings, and an (active) open to change. For example, the focus can be a critical attitude towards research, why this is considered productive, and how to assume this critical mindset. This will arm researchers with an intrinsic motivation to conduct good research, aka research using only acceptable, healthy research practices.

How will this be accomplished? The changes in both knowledge and research will be implemented through education, in the form of lectures with corresponding workshops, and seminars. The lectures will discuss one or more good research practices, one or more faulty research practices, or a research attitude. These initiatives will be organized by the UvA, and are intended for UvA employees working in research or research-related practices (though people from outside the UvA are not excluded from partaking). A certain amount of attendance will be required for doing research-related work at the UvA. Finally, a small entrance fee will be charged to help finance these innitiatives.

This is how educating researchers is expected to help reduce faulty research practices, such as questionable research practices and fraudulent behaviors, in psychology. It should be clear that this solution does not pose the previously discussed changes to the system are bad, or indeed unnecessary, but rather that changing the system would not be very effective unless the researcher is guided through some changes as well. Hopefully, these changes will not only mend the reputation of psychology as a science, but also restore its scientific value and integrity.



The Sinking Ship of Science: Suggestions for improvement of the peer-review system

I have written a letter to Dr. Eric Eich, editor in chief of Psychological Science. I have proposed a suggestion for improvement of the current peer-review system. First I outlined the problems associated with the current system. Peer-review is underappreciated in the current system: it is often seen as a burden and has to be done in one’s spare time, and is not much rewarded. Also, the author often does not know who the reviewer is. Therefore, it is easy for the reviewer to not spend a lot of time on the review, or even be plain mean. Also, the lack of time spent on the review, makes it more likely for studies in which the researchers have engaged in questionable researchers to slip through.

I have proposed a new peer-review system. In this system, the reviewer does not know who the author is, to prevent certain biases (such as the status of the author, whether he has published many articles in high-impact journals, et cetera). Reviewers receive rewards in the form of review-credits. The authors assigns 1-4 review-credits to the reviewer, based on the quality of the feedback and the usefulness of the review. The editor can then subtract or add one research credit, based on whether the review contains a clear recommendation for the editor concerning publication. The reviewer eventually thus ends up with 0-5 review-credits.

These credits become publicly available, at first on a website but could perhaps also become visible in search engines, next to the number of publications a researcher has. This is a scientist’s review index, with the total number of credits, the number of articles reviewed, and the average number of credits obtained per article. Universities may eventually consider including reviewing in the job descriptions and thus paying for reviews, as good reviewers have a high status.

The review-index will result in reviewers doing a better job on reviews. This results in higher quality of published research articles, as questionable articles are less likely to slip through. Therefore, also authors need to pay attention to clear writing and the prevention of QRPs. Also, the overall review quality will be enhanced, as extrinsic motivation is stimulated. This way, the ship of science will be prevented from sinking.

Educating the Educators

In this essay, I argue that the scientific world would benefit from a better research education for both students and researchers. Within the University of Amsterdam, re-introducing the so called ‘Research Practical’ is a very practical and easy-to-implement way to acquaint students with several problems in science, such as lack of replication. Also, in this course students can learn how to deal with things like non-replication and finding of a null effect.

The university should also take care to educate their researchers. This can be done through courses in methodology and statistics, or through an open forum on which researchers can discuss their methodological and statistical considerations with their colleagues. Educating researchers does not only benefit science: since mentors have a great influence on their pupils, good researchers will make more good researchers. And if there is one thing that science needs, it is good researchers.

Reasons to implement replication to the Research Masters Program.

As you may know, psychology has been in a state of crisis in the last years. Social psychology, for example has been shaken by several cases of scientific fraud and there have been other allegations of ‘bad research’. However, the problem of ‘bad research’ is a more general problem. In addition to fraudulent research; questionable research practices (QRPs) or the selective reporting of positive results have broadly distorted the interpretation of scientific data. In some cases, the distortion between what the data are and what the data are interpreted to be is so large that current research cannot be used by the private sector. For example, the pharmaceutical company Bayer reports that they are unable to use ‘scientific findings’ from universities, stating that sometimes as many as 75% of the results cannot be reproduced. This means that some of the ‘knowledge’ that universities produce is inaccurate or false, too unreliable to develop new medication.

However, psychology needs reliable results, if we want to apply psychology to gian something from it. Therefore, I think it is necessary and urgent that universities start to to implement replication into psychology via education. We are likely to need a lot of researcher that can replicate in the future, because many studies indicate that replication rates in psychology and other sciences are low (see Begley and Ellis, 2012). Psychology needs an overhaul in order to advance as a science and therefore it needs a lot of researchers who are able to adequately replicate studies. However, they will not appear out of nowhere. If we do not educate people, we risk that psychology gets a bad reputation among scientists and it may hurt the ties to the private sector. The private sector will not invest in a field that cannot produce applicable, reliable science, which will affect funding for psychological research negatively in the long run.

In order to make the process of reviewing and advancing our science easier, universities should raise a new generation of scientists that is able to increase the reliability of scientific findings by replication. This can only happen if universities provides the means to educate students who are willing and capable to do this. Replication is a corner stone of science and formal education is needed to do it.

Begley, C. G., & Ellis, L. M. (2012). Raise standards for preclinical cancer research. Nature, 483, 531–533.

Use and misuse of p-values; A review of NHST

In my final essay I present a case against null hypothesis significance testing using p-values. I first present several – by far not all – drawbacks and misinterpretations regarding NHST and its results. Then I present an example of a potential problem with using p-values as a decision criterion. My letter ends with a very brief summary of several methods that can be presented next to- or used as alternatives for NHST. Methods include; confidence intervals, effect sizes, and bayesian statistics.

The goal of this overview is to make people aware of controversies in this part of scientific practice, and perhaps to inspire a new view on how we should be educating new scientists. By offering not only theories based on p-value significance testing, but also discuss its flaws and alternatives.



Changing the culture of social science

Research is vulnerable for human factors like the tendency to find results in line with our predictions or falling for the temptations of status by for instance make your data prettier than it actually is. These problems can be partly prevented by using stricter policies like Open Science Framework and preregistration. However, they won’t change everything because it creates a ‘police culture’, it solely focuses on extrinsic motivation. These implementations do not take into account intrinsic motivation, something that could be very useful in achieving high quality research. Therefore, to protect social science from these human factors, social control is needed to make researcher critical and committed towards their own behaviour. Because these social control mechanisms are more or less embedded in the cultural environment, I analysed the cultural environment of social science in my essay. I described that this cultural environment is pretty much individualistic; researcher work in isolation, a displacement of goals by striving to improve a higher status instead of achieving high quality research. Furthermore, fear, a lack of openness, integrity and collaboration can be observed in this culture. In my advice I advocate implementing a more collectivistic culture in which social control mechanism are present. I focused on three concrete implementation. First, integrity meeting every year to refresh the purpose of doing science and help researcher do not lose track of this goal by for instance discussing how to deal with publication pressure. Second, peer-coaching to promote collaboration, openness within and between departments. Researchers are required to discuss difficulties and mistakes they encounter and give advice to each other. Third, education about methods and statistics for staff, new staff (including Phd’s) and research master students. Due to the dynamic of science education is needed to keep researcher up to date how to correctly analyse data. With these implementation solidarity, openness and integrity are created, changes that will improve the quality of social science.

The Cure of Pre-registration in Psychology

In my essay I described some very important problems within the field of psychology such as publication bias, lack of replication and QRP’s such as p-hacking and HARKing. All these problems can be addressed and possibly resolved through pre-registration. Pre-registration is the practice in which researchers defined their research method and hypnotize before conducting their research. It prevents publication bias because results no longer define whether studies are published publication. Second it prevents QRP’s such HARKing and p-hacking because researcher are no able to play around with the data and hypothesis and finally it promotes more replication studies. 

Pre-registration can be a solution to many problems in psychological science and should be accepted as a standard practice in psychological research. This can only be accomplished if both researchers and publishers participate. First, journals need to accept pre-registered studies. Second, researchers should want to publish in such journals. This last part can be accomplished by funding organizations such as NWO. If only researchers  who  pre-registered their work are funded, then many researchers will accept the practice of pre-registration. This way, pre-registered studies could address the current problems in psychological science.

Programme Zwaartekracht

In my essay I wrote a letter to Jet Bussemaker. For those of you who don’t know her, she’s the minister of Education, Culture & Science (OC&W). The ministry of OC&W in colloboration with the Dutch Organization of Scientific Research (NWO) have recently set up the programme “Zwaartekracht”. More information can be found at the NWO website:
Though this programme has certain benefits, such as the fact that it’s long term funding (you can get up to 10 years of funding), it also has many problems. Especially in the light of the current debates which are the topics of this course (fraud, questionable research practices, replication). In my essay I critically examine the criteria use in the programme “Zwaartekracht” to judge who gets funding and who doesn’t. These criteria are quality of involved scientists (weight: 30%), quality of the scientific research programme (weight: 35%), Institutional integration and organizational structure of the consortium (weight: 25%) and lastly the applicability of gained knowledge (weight: 10%). My critique of the first criterium is mainly how do you measure the quality of a scientist? I’ve assumed that it’s probably by using their H-index. I then describe some problems with using the H-index to measure quality.The main point being that you can use QRPs to artificially “improve” a paper and paradoxically at the same time improve your quality as a scientist as measured by the H-index. My critique of the second criterium mainly focuses on the fact that they completely new research that will lead to new insights and international breakthroughs. When of course, you can not know the results of your study before you’ve conducted it especially when it’s something completely new which no one has ever done before. There are ways that you can know the results such as fabricating data or using QRPs (specifically presenting exploratory results as confirmatory) to make your data say whatever you want. When discussing the 3rd criterium I focus on the fact that in the description of this criterion it is mentioned that they programme should contribute to the education of young and talented researchers. Since this offers the opportunity to integrate education about research ethics and integrity and mentoring of researchers I focused on the work done by Melissa Anderson to explain why this kind of education is crucial. The 4th criterion is only shortly addressed in the way that it’s hard to find applications for new and groundbreaking research. Plus this focus on application can be a reason why there are no projects from the Humanities or Social Sciences that received funding by the programme Zwaartekracht . It can be harder to find good concrete applications for the research findings of these disciplines. As Han van der Maas’ famous example shows psychology has yet to invent a fridge.

I conclude that the programme “Zwaartekracht” is a clear example of the current rewarding and funding system, which encourages the use of QRPs and discourages good practice such as replication and pre-registration. Solutions for this are to require everyone who gets funded by the ministry or NWO to pre-register and to pay special attention to the education of scientists on these topics. Another option would be to introduce a special programme to fund replication studies in order to ensure that new insights and international breakthroughs hold up and are not simply too good to be true. Together these measures could improve the quality of scientific research and change the current research climate into a climate in which good science is rewarded and bad science punished instead of the other way around.