Why do some psychology studies not replicate well?

Introduction

In 2010, an article from the Open Science Collaboration sent shockwaves through the psychology world when it revealed that only about a third of 100 psychology experiments found the same results as the original studies. A third! As you might imagine, some psychologists were quite concerned about this, going so far as to call it a “crisis”. Today, we often hear about a “replication crisis” in psychology, but how bad are things really? Are we all doomed to produce unreliable research? Well, I’m not so sure we are.

If you want some classic examples of studies that have fallen victim to the replication crisis, you need look no further than my own field of nonverbal psychology. It seems that many of the famous studies here have not replicated well at all, so are they truly bad experiments or is something else going on? Let’s look at some examples.

The Facial Feedback Hypothesis

A famous study by Strack et al (1988) told us that when you hold a pen between your teeth (forcing you to smile), you may find a newspaper cartoon funnier than when you hold it between your lips (forcing you to frown). This, they claimed, was evidence of the Facial Feedback Hypothesis; the idea that our facial expressions can actually drive our emotions.

This study appeared to be a great example of the power of the human mind, though some were sceptical. In 2016, Wagenmakers et al. took the wind out of these sails with a comprehensive replication study that failed to find this result. Many believe this single study spelled the end of the Facial Feedback Hypothesis and it was often cited as an excellent example of the importance of running replication studies. However, there was a problem.

Although Wagenmakers and his team used the same materials as the original study, they didn’t quite replicate the experiment. Critically, in their version, they placed a camera in front of the participants and filmed them as they took part. Naturally, this was slightly concerning to some psychologists as stacks of research has found that people tend to act much differently when they’re being filmed.

This deviation caught the attention of Noah et al (2018) who decided to run two versions of the experiment; one, a true replication (without the camera present) and the other, the failed replication (with a camera). Not surprisingly, they found support for both studies; that, when you don’t have a camera present, you see the effect as normal but, when you do, the effect disappears. As a result, it seems we were a bit hasty to write off the original findings.

Power Posing

A similar story was found with ‘power posing’. As I explained in a previous blog post, Carney at al (2010) made waves in the psychology world when they claimed that sitting or standing in a power pose for one minute could make you feel more powerful, change your behaviour, and even affect your hormonal levels.

Enter the replication study by Ranehill et al (2015) and the controversy truly began. With a larger sample and a more even gender split, they found that power posing had no effect on their participants’ behavioural or hormonal responses. Once again, people cited this as the beginning of the end for power posing research. However, once again, there was a problem.

As before, this was not really a true replication. While the original authors were careful not to tell participants what the purpose of the study was (by coming up with a clever cover story about why they needed to be in different poses), the authors of the replication study told them openly. Also, instead of holding the pose for one minute, their participants held them for six.

Interestingly, while many often cite the failure to observe behavioural and hormonal effects in the replication study as evidence against power posing, they miss one key finding; that power posing still made their participants feel more powerful. A robust finding which holds up well today.

Conclusion

It’s very important that we run replication studies in psychology, but it’s also important that we examine these replication studies through the same lens as the originals. Science is an ongoing pursuit of knowledge and that doesn’t stop when we find that the results of a study do not replicate. Instead of dismissing the original findings, we need to work together to understand why they were found and where we go next. That’s the heart of collaborative science.

Author

Senior Lecturer in Psychology, Dr Daniel Gurney