Changing the culture of social science

Research is vulnerable for human factors like the tendency to find results in line with our predictions or falling for the temptations of status by for instance make your data prettier than it actually is. These problems can be partly prevented by using stricter policies like Open Science Framework and preregistration. However, they won’t change everything because it creates a ‘police culture’, it solely focuses on extrinsic motivation. These implementations do not take into account intrinsic motivation, something that could be very useful in achieving high quality research. Therefore, to protect social science from these human factors, social control is needed to make researcher critical and committed towards their own behaviour. Because these social control mechanisms are more or less embedded in the cultural environment, I analysed the cultural environment of social science in my essay. I described that this cultural environment is pretty much individualistic; researcher work in isolation, a displacement of goals by striving to improve a higher status instead of achieving high quality research. Furthermore, fear, a lack of openness, integrity and collaboration can be observed in this culture. In my advice I advocate implementing a more collectivistic culture in which social control mechanism are present. I focused on three concrete implementation. First, integrity meeting every year to refresh the purpose of doing science and help researcher do not lose track of this goal by for instance discussing how to deal with publication pressure. Second, peer-coaching to promote collaboration, openness within and between departments. Researchers are required to discuss difficulties and mistakes they encounter and give advice to each other. Third, education about methods and statistics for staff, new staff (including Phd’s) and research master students. Due to the dynamic of science education is needed to keep researcher up to date how to correctly analyse data. With these implementation solidarity, openness and integrity are created, changes that will improve the quality of social science.

The Cure of Pre-registration in Psychology

In my essay I described some very important problems within the field of psychology such as publication bias, lack of replication and QRP’s such as p-hacking and HARKing. All these problems can be addressed and possibly resolved through pre-registration. Pre-registration is the practice in which researchers defined their research method and hypnotize before conducting their research. It prevents publication bias because results no longer define whether studies are published publication. Second it prevents QRP’s such HARKing and p-hacking because researcher are no able to play around with the data and hypothesis and finally it promotes more replication studies. 

Pre-registration can be a solution to many problems in psychological science and should be accepted as a standard practice in psychological research. This can only be accomplished if both researchers and publishers participate. First, journals need to accept pre-registered studies. Second, researchers should want to publish in such journals. This last part can be accomplished by funding organizations such as NWO. If only researchers  who  pre-registered their work are funded, then many researchers will accept the practice of pre-registration. This way, pre-registered studies could address the current problems in psychological science.

Programme Zwaartekracht

In my essay I wrote a letter to Jet Bussemaker. For those of you who don’t know her, she’s the minister of Education, Culture & Science (OC&W). The ministry of OC&W in colloboration with the Dutch Organization of Scientific Research (NWO) have recently set up the programme “Zwaartekracht”. More information can be found at the NWO website: http://www.nwo.nl/financiering/onze-financieringsinstrumenten/nwo/zwaartekracht/zwaartekracht.html.
Though this programme has certain benefits, such as the fact that it’s long term funding (you can get up to 10 years of funding), it also has many problems. Especially in the light of the current debates which are the topics of this course (fraud, questionable research practices, replication). In my essay I critically examine the criteria use in the programme “Zwaartekracht” to judge who gets funding and who doesn’t. These criteria are quality of involved scientists (weight: 30%), quality of the scientific research programme (weight: 35%), Institutional integration and organizational structure of the consortium (weight: 25%) and lastly the applicability of gained knowledge (weight: 10%). My critique of the first criterium is mainly how do you measure the quality of a scientist? I’ve assumed that it’s probably by using their H-index. I then describe some problems with using the H-index to measure quality.The main point being that you can use QRPs to artificially “improve” a paper and paradoxically at the same time improve your quality as a scientist as measured by the H-index. My critique of the second criterium mainly focuses on the fact that they completely new research that will lead to new insights and international breakthroughs. When of course, you can not know the results of your study before you’ve conducted it especially when it’s something completely new which no one has ever done before. There are ways that you can know the results such as fabricating data or using QRPs (specifically presenting exploratory results as confirmatory) to make your data say whatever you want. When discussing the 3rd criterium I focus on the fact that in the description of this criterion it is mentioned that they programme should contribute to the education of young and talented researchers. Since this offers the opportunity to integrate education about research ethics and integrity and mentoring of researchers I focused on the work done by Melissa Anderson to explain why this kind of education is crucial. The 4th criterion is only shortly addressed in the way that it’s hard to find applications for new and groundbreaking research. Plus this focus on application can be a reason why there are no projects from the Humanities or Social Sciences that received funding by the programme Zwaartekracht . It can be harder to find good concrete applications for the research findings of these disciplines. As Han van der Maas’ famous example shows psychology has yet to invent a fridge.

I conclude that the programme “Zwaartekracht” is a clear example of the current rewarding and funding system, which encourages the use of QRPs and discourages good practice such as replication and pre-registration. Solutions for this are to require everyone who gets funded by the ministry or NWO to pre-register and to pay special attention to the education of scientists on these topics. Another option would be to introduce a special programme to fund replication studies in order to ensure that new insights and international breakthroughs hold up and are not simply too good to be true. Together these measures could improve the quality of scientific research and change the current research climate into a climate in which good science is rewarded and bad science punished instead of the other way around.

A call for an open-source neuroscience

Hi all, I wrote my final project as an advisory letter to a leading neuroscientist at the University of Amsterdam, in which I argue for a fully open-source environment in neuroscience research. I will summarize my main points in this short blog post.

A call for open-source neuroscience

Neuroscience is hard. It involves a lot of data, a lot of (pre)processing, and consequently a lot of programming. The average neuroscientists uses a variety of software packages, which are used in different parts of the research process (stimulus delivery, preprocessing, analysis, visualization) and can add up to hundreds of lines of code. This also means that a mistake is made easily, and importantly, unnoticed. Other fields, in which “big data” is common currency, such as artificial intelligence and computer science, have taken appropriate measures to tackle these risk factors. One of these steps is the (almost) complete shift from proprietary to open-source software. The advantages that this change brings, summarized in the themes qualitytransparency, and uniformity, and will be discussed shortly.

Code quality is improved through the active and involved user community centered around open-source software. This community ensures quality through an emergent community-driven quality control and, moreover, drives innovation. Successful examples of open-source projects in science are the extensive open-source machine learning library Scikit-learn (scikit-learn.org) and the statistical software package “R” (r-project.org), which both have grown into widely used programs in just a couple of years.

Transparency, and consequently reproducibility, are improved through adoption of open-source software as it enables complete code documentation. Proprietary software, especially when they are GUI-based, on the other hand hamper transparency and reproducibility, (1) because their source code is unavailable, and (2) because potential replicators may not possess the original research’s software (which can be extremely expensive).

Uniformity in computing environments will further improve transparency and reproducibility. This feat is, however, only possible in an open-source framework, in which software can be accessed, modified, and suited to the neuroscientist’s needs (see figure 1 below).

opensource2

Figure 1Left diagram: An open-source environment allows for development of different programs for differnent purposes (for example Freesurfer for cortical parcellation and FSL for modelling of neural activity). These programs can be implemented in a common platform such as Nipype because of their open-source structure. This way, studies can easily adopt a fully open-source analysis pipeline, rendering replication and comparison easier. Right diagram: Proprietary software packages are all developed in their own computational environment and cannot be integrated in an overarching program such as Nipype. As a consequence, it leads to studies that are more difficult to replicate and compare to each other.

In summary, open-source provides several advantages over proprietary software in terms of quality, transparency, and uniformity. Currently, open-source offers plenty, high-quality alternatives to today’s proprietary standards in neuroscientific research, but changing the mind-set of “old-fashioned” neuroscientists should be priority.

 

The Golden Road of Open Access is the Path of Least Disruption

A few days ago, the Netherlands Organisation for Scientific Research (NWO) – the country’s major funding agency – announced that it will soon start to require funded research to be published in a fully open access journal. At a symposium at the University of Amsterdam to mark the beginning of the open access week, NWO president Jos Engelen today reasserted this aim and set out a bold vision: in ten years, all research in the Netherlands will be published following this ‘Golden Road’ to open access.

It would be easy to mistakenly believe that such a commitment to the ‘Golden Road’ to open access is a revolutionary step. After all, big words about making knowledge freely accessible to the public are at play. But the opposite is true: the seeming gold standard is perhaps the most conservative implementation of open access currently discussed, and misses an opportunity for profound changes to how scientists collaborate and scientific findings are communicated.

The cost of gold

In simple terms, the Golden Road states that scientific output should be published in fully open access journals. At today’s symposium, it was mainly contrasted with the ‘Green Road’, in which output is deposited in open access repositories and may still be published in closed access journals.

Golden Road open access has seen significant success in the Netherlands, growing two percentage points annually over the last few years and garnering the support of state secretary for Education, Culture and Science Sander Dekker. But this success has come at the expense of Green Road repositories, as Wouter Gerritsma of Wageningen University showed using the example of Narcis, the national open access repository. That is not only costly; it is also to the detriment of grey literature such as Ph.D. dissertations. Traditional papers make up only 45% of the content on Narcis; but the remaining 55% remain hidden when gold, rather than green, is the publication standard. Indeed, grey literature benefits particularly strongly from open access repositories: seven out of the top ten publications downloaded from UCL’s ‘Discovery‘ repository are dissertations, according to library director Peter Ayris.

The Golden Road also often implies an ‘author pays’ funding model, in which papers are freely accessible, but authors pay the journal for publication. These costs can be significant: a paper in one of the PLoS journals can cost between $1,350 and $2,900. In contrast, repositories – which do not provide peer review – are cheap to run and free to use. Gerritsma that the cost of ‘fully gold open access’ in the Netherlands at current output numbers and publication costs (around $1,200 per paper) would cost $27.7 million – not significantly less than the $34 million Dutch universities currently pay for journal subscriptions. Worse, still: because universities would have to pay for both open access publishing and closed access subscriptions during an unspecified transitory phase, they would face both bills at the same time.

A missed opportunity for change

One of my favourite quotes about the open access movement states that “if open access does not hurt Elsevier, we are doing it wrong”. The Golden Road seems to be exactly that wrong path: it merely shifts around when the bill is paid, but the sums remain (approximately) the same. This would perhaps seem reasonable if the cost of journal subscriptions reflected the value rendered to the scientific community by publishers, but clearly they don’t: scientific publishing is one of the most profitable industries there are, and the cost of subscriptions is widely considered exploitative.

The Golden Road also misses an opportunity to replace the century-old publication model based on journals with procedures and technology for the digital age. At a recent question-and-answer session, Denny Borsboom called for psychology to adopt the publishing model of mathematics and physics, in which results are upload onto arXiv and receive open peer review there, the subsequent publication in journals almost being secondary. In that, he echoed Brian Nosek, who (together with Yoav Bar-Anan) has laid out a grand agenda for opening up scientific communication which gradually decouples steps such as evaluation and publication, which now seem inextricably linked. Open access is a first step in this agenda – but if open access publishing remains firmly in the hands of for-profit journals, the following steps may never come.

Addendum: Wouter Gerritsma’s presentation is now only here.

Discussing good science on Twitter

A while ago I wrote a blog post here on Using Twitter to Explore the Frontiers of Psychological Science, in which I showed how students and researchers can harness social media to discuss good science.

Twitter is a great platform to gather information, discuss ideas, and spread your output. As a discursive medium, it allows anybody quick and barrier-free access to contemporary debates. It is also a powerful tool to spread the word about your own work and the work of others.

To give an example, I have summarised how I have used Twitter to write and publicise two of my posts on this blog. Check it out on Storify!

“You get people who are great with statistics who should never talk to any sample besides their Phd students who are equally crazy” – Kai Jonas

This Thursday we had an interesting discussion with Kai Jonas and Coosje Veldkamp on QRPs, here are the most interesting bits of the dicussion according to us.

The discussion started with Coosje giving quite a different interpretation about what QRPs are than what we discussed in class so far. They thought that what we call QRPs could better be called ‘questionable statistical practices’ and that the term QRP should be used for practices that have to do with more ethical aspects of research, such as the way you treat your participants. Kai agreed and mentioned that what constitutes as a QRP depends on the paradigm a scientific field is using at the moment. What is considered to be wrong changes over time and along with paradigm shifts come shifts in focus: currently we are focused on the statistical side of doing proper research but depending on what is ‘fashionable’ (or in what area scandals are uncovered) this focus will change to a different aspect of research, such as ethical questions (the Facebook study that Kai mentioned).
Both Coosje and Kai thought a narcissistic personality could make one more prone to committing QRPs, but only if the person is aware they are cheating. Narcissists are less critical of themselves in general so they would be able to look themselves in the eye for longer after committing a QRP. However, both also thought a lot of the times the cause of QRPs is just plain ignorance and that they are not committed intentionally.

If we consider QRPs to be ethical practices, Kai thinks it’s debatable what is wrong and what is not. However, statistical decisions are more strict. Coosje and Kai agree that the most important part is reporting everything you do and all decissions that you make. Since this can make your article somewhat unreadable they suggest supplements on the webpage of the journal, appendices and places like open science framework to report some (but of coure not all) of this information.
Also, people should have a decent knowledge of what they are doing, but not every psychologists needs to be a statistician. This is not possible. One can learn about fields linked with one’s own, such as statistics, but one should not forget one’s core competence (in this case: psychology). At this moment we treat people too much like lab rats, while the core competence of a psychologist is that he or she can link him- or herself with other people (Kai).

On one hand, it was thought to be a good idea if every research team had its own statistician. This person would be able to run all the analyses correctly and this would reduce the amount of statistical errors. However, even among statisticians there is no consensus on every topic or analysis and not all statisticians understand everything. On the other hand, having multiple people analyse the data was thought to be a good way to reduce statistical errors. Coosje told us that in her research group all the analyses are run by multiple people and if there are discrepancies in the outcomes of different peoples these are discussed. The best would be a combination of these. However, this is hard to achieve: every research group would need at least two statisticians and at least two theoretical people (to provide the theoretical background and make sure there are no terrible interpretation errors) in order for intersubjectivity to work.

Both Kai and Coosje were asked which factors could influence whether people are prone to engaging in QRPs. Kai mentioned that what constitutes a QRP depends on the paradigm a scientific field is using at the moment. What is considered to be wrong changes over time and along with paradigm shifts come shifts in focus: currently we are focused on the statistical side of doing proper research but depending on what is ‘fashionable’ (or in what area scandals are uncovered) this focus will change to a different aspect of research, such as ethical questions (the Facebook study that Kai mentioned).

Both Coosje and Kai thought a narcissistic personality could make one more prone to committing QRPs, but only if the person is aware they are cheating. Narcissists are less critical of themselves in general so they would be able to look themselves in the eye for longer after committing a QRP. However, both also thought a lot of the times the cause of QRPs is just plain ignorance and that they are not committed intentionally.

As Kai is editor of the first preregistration-only journal in psychology at this moment, it was of course interesting to ask him something about this subject. The possibility that someone else steals your ideas if you preregister your study came up. Kai answered that the controlling mechanism that preregistration has is that it you know who reviews your manuscript. Therefore, if someone was to come up with the same idea you know that he or she might have gotten it from your manuscript. However, in any case the problem is already out there. When sending out your manuscript for peer review or applying for a grant, you also put your idea out there. There are even people who “go shopping” at poster conferences. This risk of your idea being stolen is not created by preregistration. Coosje even remarked that preregistering your study might be a safeguard against it being stolen: the moment of you sending in your idea has an absolute date so this could be used as proof that you were the first one to come up with the idea.

They were also questioned about the balance between novelty and replication. Kai’s opinion was that it depends on the phase a research field is in. When starting up a new field, novel findings are fine but one should always go back to check what works and what does not. Limiting the amount of novel findings immediately is not a good idea: effects might not instantly replicate while other findings still hold up. He thinks that self-regulation will occur at one point. Coosje added that people should also be careful not to convince themselves too quickly. Researchers should always try to replicate their own research before accepting is as being the truth. Replication might be very time-consuming but it’s definitely worth it.