Judging the credibility of scientific results

Despite what we teach in our courses, there is considerable disagreement about how we judge the scientific merit of research findings. One method we might consider is whether scientific results are reproducible. Separate initiatives in psychological science and cancer biology suggest, though, that reproducibility of published findings isn't great. 

It turns out that even when using "gold standard" research methods, scientists are human and susceptible to the same mistakes and biases as other people. We tend to promote findings that we want to be true (or that feel intuitive) rather than rigorously examining whether results are, in fact, replicable in the wider world (e.g., Flossing [cf];  Power-posing). The credibility crisis of research findings is not unique to psychology, but psychological scientists are at the forefront of refining research practices throughout the life sciences.

One of the most vocal solutions to dealing with this reproducibility crisis is embracing the principle of openness in how we conduct scientific research. By identifying the mistakes we are most prone to, methodologists in psychology and wider sciences are proposing updated methods for improving the quality and value of our scientific findings. Most of these have to do with being more transparent and open about how we got the results that we did.

Of these transparent and open methods, replication is key. Over the past 5 years there has been a groundswell of initiatives aimed at precisely repeating a study's methods to examine how readily a result can be replicated. So far, these Registered Replication Reports have examined more than a dozen different psychology results using data collected from labs around the world. Last year our Social Cognition Lab at Massey was one of 23 labs from around the world trying to replicate a result known as the Ego Depletion Effect. Although there have been hundreds of peer reviewed research articles advocating Ego Depletion Effect over the past two decades, there was little evidence of the effect across the 23 labs in our replication. We can only imagine how many other non-replications have gone unreported over the last two decades because everyone just assumed Ego Depletion was an easily reproduced effect.

This, then, puts the burden of proof back on advocates of the theory–if none of these researchers can reproduce the effect, is it really a credible effect? Should we really write about it in textbooks? Should President Obama really have out-sourced his suit choices? Reproducible, transparent practices define scientific research. If we can't replicate an effect or look behind the curtain to see how an effect comes to be, we are doing something closer to parlour tricks than science.