Often researchers have te deal with missing data in psychology and social sciences. Missing values have to be dealt with because most statistical analyses are not designed for missing data. At the moment most of the methods often used to handle missing data have a lot of problems including biased results. Therefor they are not recommended to use. Some examples of these methods are listwise deletion, pairwise deletion and mean imputation/replacement.
Luckily there are methods that can be used and have less of these problems. In this blog two of them will be discussed: multiple imputation and maximum likelihood.
With multiple impuation the distribution of the variable with missing data is estimated through the observed data. When this distribution is estimated a new dataset is created with the missing values replaced by random drawn values from the estimated distribution. But when only one dataset is made one assumes that the estimated distribution is the same as the population distribution. This is often not the case and will give an underestimation of the standard error. To tackle this problem more datasets are made. When all these datasets are made it is possible to calculate a pooled mean and standard error. Finally with this pooled mean and standard error the analyses can be performed.
Maximum likelihood is a more complicated method for handling missing data. With this method missing data is not impuated but it uses the observed data of a participant with missing values to correct the parameters used in a model. This is done with a maximizing function. So although missing values are not replaced with an estimate of what the missing value should be, the observed data of a participant is still used in the estimation of the model parameters. This looks similar to multiple imputation but the difference is that no new dataset is created and then the analysis is done but the maximum likelihood method is used together with the analysis. The advantage of this is that produces accurate standard errors because the sample size is the same. Which is not the case with the pooled means and standard errors in multiple imputation. This method mainly has practical problems. It is not included in many statistical software packages and the sample size has to be rather large. This is often a problem in psychological research.
Because it in psychological research sample sizes are often small it is probably better to use the multiple imputation method. It is important to educate researchers about this methods and about how to report missing data. But there is also a responsibility for statistical software developers to make methods like multiple imputation and maximum likelihood more accessible. Furthermore it is suggested to not make listwise or pairwise deletion the default method in handling missing data in statistical software.
Donders, A. R. T., van der Heijden, G. J. M. G., Stijnen, T., & Moons K. G. M. (2006). Review: a gentle introduction to imputation of missing values. Journal of Clinical Epidemiology, 59, 1087-1091.
Enders, C. K., Bandalos, D. L. (2001). The relative performance of full information maximum likelihood estimation for missing data in structural equation models. Structural Equation Modeling, 8, 430-457.
Say you, as a scientist, want to predict something in your research, such as the amount of oxygen someone can uptake. You would want to have certain measures that could say something about that, such as a person’s age, height and weight. With (some of) these predictive measures, or predictors, you would then want to try and find out whether you can actually predict something about how much oxygen someone can uptake. To this end, the method of stepwise regression can be considered.
There are two methods of stepwise regression: the forward method and the backward method.
In the forward method, the software looks at all the predictor variables you selected and picks the one that predicts the most on the dependent measure. That variable is added to the model. This is repeated with the variable that then predicts the most on the dependent measure. This little procedure continues until adding predictors does not add anything to the prediction model anymore.
In the backward method, all the predictor variables you chose are added into the model. Then, the variables that do not (significantly) predict anything on the dependent measure are removed from the model one by one.
The backward method is generally the preferred method, because the forward method produces so-called suppressor effects. These suppressor effects occur when predictors are only significant when another predictor is held constant.
There are two key flaws with stepwise regression. First, it underestimates certain combinations of variables. Because the method adds or removes variables in a certain order, you end up with a combination of predictors that is in a way determined by that order. That combination of variables may not be closest to how it is in reality. Second, the model that is found is selected out of the many possible models that the software considered. It will often fit much better on the data set that was used than on a new data set because of sample variance.
There are no solutions to the problems that stepwise regression methods have. Therefor it is suggested to use it only in exploratory research. Stepwise regression methods can help a researcher to get a ‘hunch’ of what are possible predictors. This is what is done in exploratory research after all.
But off course confirmatory studies need some regression methods as well. Luckily there are alternatives to stepwise regression methods. One of these methods is the forced entry method. In this method the predictors are put in the model at once without any hierarchical specification of the predictors. For example, a scientist wants to test a theory in which math ability in children is predicted by IQ and age but he has no assumptions about which is the best predictor. In this case the forced entry method is the way to go.
Although the forced entry method is the preferred method for confirmatory research by some statisticians there is another alternative method to the stepwise methods. This is the hierarchical (blockwise entry) method. Basically, this method consists of two steps. In the first step predictors are entered in the model in a hierarchical manner. For example, a scientist specifies a model in which math ability is best predicted by IQ and than by age. Step two is an optional step in which the scientist can add more predictors. These predictors can be entered in the model hierarchical, forced entry or stepwise. Of course the problems mentioned earlier still occur when the stepwise methods are used in the second step.
In the end all methods can have a purpose but it is important for a scientist to know when to use the right method for the right purpose.
Lucas & Jochem
In science there are some researchers whom formulate or change their hypothesis after they have seen the results of their statistical analyses, also called HARKing. I want to point out a few options why some researchers show this behaviour. Maybe they are willing to change or formulate their hypotheses post hoc because they need to publish and therefor they need a positive significant result, because of publication bias. Another option is that they are ignorant to the fact that HARKing can be harmful to science and see no harm in doing it. But why is HARKing bad for science?
Because with HARKing a hypothesis is made after the results are known the chance of falsely rejecting a null-hypothesis increases, this means an increase of type I errors. Furthermore there will be a distorted image of effect sizes because effect sizes found will be larger than the true effect sizes. Another reason why HARKing is bad for science is that it increases the chance of wasting resources like time and money. This is because there are more studies that have to be replicated that have no true effect.
Luckily there are some good remedies against HARKing. For example, replication of research makes it possible to find if there was any HARKing. If ignorance of the consequences is the reason that some researchers HARK then education would be a good solution as well. But pre-registering of research is probably the best remedy for HARKing because this really minimizes the possibility to HARK.
Kerr, N. L. (1998). HARKing: hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196-217.