Once every now and then, we come across an article that strikes us as particularly strange. For me, Harris et al. (1999), investigating the effect of intercessory prayer on patients in a hospital’s cardiovascular and coronary unit, is such an article. Not just because they devised an arbitrary outcome measure, but because of patterns in their reported data.
Typically, these data should differ a bit. Perhaps not much if the measurement is very precise and the sampling and randomization procedures worked exceptionally well, but they should differ at least a bit. Harris et al. report six rather suspect standard errors (SE) of the means (M) for three measurements (there were two groups): the SE pairs are [.27, .26], [.1, .1] and [.009, .008]. I was puzzled, mainly because the two groups were of different size. Generally, the larger the sample size, the smaller the standard error.
I set out to test whether this conspicuous coincidence could be expected on the basis of chance. Simonsohn (2012) provided a bootstrap-method to test for this, taking into account means and standard deviations. The values presented in this particular paper did not produce a significant result, at least not by using this method.
I looked at two more articles that list Mr. Harris as co-author (Duda et al., 2009; Skulas-Ray et al.,2011), incidentally, both investigate the effect of omega-3 fatty acids on animals (one on human, the other on non-human animals – rats). By the way, Mr. Harris would probably take issue with the previous sentence, as he is a vocal proponent of Intelligent Design. Of course, as long as what is being referred to as ‘intelligently designed’ is a scientific study, I have no trouble promoting the same. But I tend to doubt an omega-3-researcher’s integrity if said researcher seems heavily invested in the testing of blood omega-3 levels, such as Mr. Harris, who is president and CEO of a certain company called Omegaquant.
I must admit that I am not very knowledgeable in the field of medicine, so perhaps I missed something important – if so, please let me know, but the Simonsohn analysis of these two papers gave interesting results.
Duda et al. produced a table of results for 13 variables, 4 of which seemed suspect to me, because they had virtually (and in one case de facto) identical standard errors across 8 conditions. However, only the de facto identical values produced a significant p-value for Simonsohn’s method: out of 100000 simulations, not a single one generated data similar or more extreme than Duda et al.’s. As it happens, this is the variable the authors used to test their main hypothesis.
The last article, Skulas-Ray et al. (2011), was truly something new for me. In a first table, they present 25 variables and 3 measurement points—all standard errors are identical across conditions and in 3 cases even across variables. Perhaps unsurprisingly, the Simonsohn method reports all of these data to be unlikely with p=0 for 10000 simulations. In a second table, out of 18 new variables, only 4 have instances of deviation from identity across conditions and all of these deviations are minimal. I doubted my own results so much that I started looking at similar papers with similar methodology, but patterns in the data of those articles are not similar to that degree (Stirban et al., 2011; Goodfellow, Bellamy, Ramsay, Jones and Lewis, 2000), so I am left wondering.
Either there is something seriously wrong with the authors’ data, my implementation of Simonsohn’s method or the way I applied it to the data. There are other alternatives, but these are the most obvious conclusions given these results. Certainly, the field of fraud detection requires a great deal more attention and research to avoid false accusations or even witch-hunts. To avoid confusion, researchers should take the initiative and simply post their raw data online.
Duda, M. K., Shea, K. M., Tintinu, A., et al. (2009). Fish oil, but not flaxseed oil, decreases inflammation and prevents pressure overload-induced cardiac dysfunction. Cardiovascular Research, 81, 319-327.
Harris, W. S., & Calvert, J. H. (2003). Intelligent Design: The Scientific Alternative to Evolution. National Catholic Bioethics Quarterly, 531-561.
Harris, W. S., Gowda, M., & Kolb, J. W., (1999). “A randomized, controlled trial of the effects of remote, intercessory prayer on outcomes in patients admitted to the coronary care unit”. Archives of Internal Medicine, 159, 2273–2278.
Goodfellow, J., Bellamy, M. F., Ramsey, M. W., Jones, C. J., & Lewis, M. J. (2000). Dietary supplementation with marine omega-3 fatty acids improve systemic large artery endothelial function in subjects with hypercholesterolemia. Journal of the American College of Cardiology, 35, 265-270.
Simonsohn, U. (2012). Just post it: The Lesson from Two Cases of Fabricated Data Detected by Statistics Alone. Available at SSRN: http://ssrn.com/abstract=2114571 or http://dx.doi.org/10.2139/ssrn.2114571.
Skulas-Ray, A. C., Kris-Etherton, P. M., Harris, W. S., Vanden Heuvel, J. P., Wagner, P. R., & West, S. G. (2011). Dose-response effects of omega-3 fatty acids on triglycerides, inflammation, and endothelial function in healthy persons with moderate hypertriglyceridemia1-3. The American Journal of Clinical Nutrition, 93, 243–252.
Stirban, A., Nandrean, S., Götting, C., et al. (2009). Effects of n—3 fatty acids on macro- and microvascular function in subjects with type 2 diabetes mellitus. American Journal of Clinical Nutrition, 91, 808-813.