ScienceDaily
Your source for the latest research news
Follow Facebook Twitter LinkedIn Subscribe RSS Feeds Newsletters
New:
  • Black Death Shaped Evolution of Immunity Genes
  • Methane-Eating 'Borgs' Taking Earth's Microbes
  • Pain Relief Without Side Effects and Addiction
  • Accounting For Dark Energy and Dark Matter
  • Meet the First Neanderthal Family
  • Physicists Confirm Hitch in Proton Structure
  • Five Hours' Sleep a Night: Disease Risk
  • Energetics of Piercing Fangs and Claws
  • Hair Straightening Chemicals and Uterine Cancer
  • Ostriches With Similar Interests Form Groups
advertisement
Follow all of ScienceDaily's latest research news and top science headlines!
Science News
from research organizations

1

2

Researchers overturn landmark study on the replicability of psychological science

Date:
March 3, 2016
Source:
Harvard University
Summary:
An in-depth examination of a 2015 landmark study showing that more than half of all psychology studies cannot be replicated has revealed serious mistakes that make its pessimistic conclusion completely unwarranted.
Share:
FULL STORY

According to two Harvard professors and their collaborators, a 2015 landmark study showing that more than half of all psychology studies cannot be replicated is actually wrong.

advertisement

In an attempt to determine the replicability of psychological science, a consortium of 270 scientists known as The Open Science Collaboration (OSC) tried to replicate the results of 100 published studies. More than half of them failed, creating sensational headlines worldwide about the "replication crisis" in psychology.

But an in-depth examination of the data by Daniel Gilbert (Edgar Pierce Professor of Psychology at Harvard University), Gary King (Albert J. Weatherhead III University Professor at Harvard University), Stephen Pettigrew (doctoral student in the Department of Government at Harvard University), and Timothy Wilson (Sherrell J. Aston Professor of Psychology at the University of Virginia) has revealed that the OSC made some serious mistakes that make this pessimistic conclusion completely unwarranted:

The methods of many of the replication studies turn out to be remarkably different from the originals and, according to Gilbert, King, Pettigrew, and Wilson, these "infidelities" had two important consequences.

First, they introduced statistical error into the data which led the OSC to significantly underestimate how many of their replications should have failed by chance alone. When this error is taken into account, the number of failures in their data is no greater than one would expect if all 100 of the original findings had been true.

Second, Gilbert, King, Pettigrew, and Wilson discovered that the low-fidelity studies were four times more likely to fail than were the high-fidelity studies, suggesting that when replicators strayed from the original methods, they caused their own studies to fail.

advertisement

Finally, the OSC used a "low powered" design. When Gilbert, King, Pettigrew, and Wilson applied this design to a published data set that was known to have a high replication rate, it too showed a low replication rate, suggesting that the OSC's design was destined from the start to underestimate the replicability of psychological science.

Individually, Gilbert and King said, each of these problems would be enough to cast doubt on the conclusion that most people have drawn from this study, but taken together, they completely repudiate it. The flaws are described in a commentary published March 4 in Science.

Like most scientists who read the OSC's article when it appeared, Gilbert, King, Pettigrew, and Wilson were shocked and chagrined. But when they began to scrutinize the methods and reanalyze the raw data, they immediately noticed problems--problems that started with how the replicators had selected the 100 original studies.

"If you want to estimate a parameter of a population," said King, "then you either have to randomly sample from that population or make statistical corrections for the fact that you didn't. The OSC did neither."

"What they did," added Gilbert, "is create an idiosyncratic, arbitrary list of sampling rules that excluded the majority of psychology's subfields from the sample, that excluded entire classes of studies whose methods are probably among the best in science from the sample, and so on. Then they proceeded to violate all of their own rules. Worse yet, they actually allowed some replicators to have a choice about which studies they would try to replicate. If they had used these same methods to sample people instead of studies, no reputable scientific journal would have published their findings. So the first thing we realized was that no matter what they found--good news or bad news--they never had any chance of estimating the reproducibility of psychological science, which is what the very title of their paper claims they did."

"And that was just the beginning," King said. "If you are going to replicate a hundred studies, some will fail by chance alone. That's basic sampling theory. So you have to use statistics to estimate how many of the studies are expected to fail by chance alone because otherwise the number that actually do fail is meaningless."

advertisement

According to King, the OSC did this, but they made a critical error.

"When they did their calculations, they failed to consider the fact that their replication studies were not just new samples from the same population. They were often quite different from the originals in many ways, and those differences are a source of statistical error. So we did the calculation the right way and then applied it to their data. And guess what? The number of failures they observed was just about what you should expect to observe by chance alone--even if all one hundred of the original findings were true. The failure of the replication studies to match the original studies was a failure of the replications, not of the originals."

Gilbert noted that most people assume that a replication is a "replica"' of the original study.

"Readers surely assumed that if a group of scientists did a hundred replications, then they must have used the same methods to study the same populations. In this case, that assumption would be quite wrong. Replications always vary from originals in minor ways of course, but if you read the reports carefully, as we did, you discover that many of the replication studies differed in truly astounding ways--ways that make it hard to understand how they could even be called replications."

As an example, Gilbert described an original study that involved showing White students at Stanford University a video of four other Stanford students discussing admissions policies at their university. Three of the discussants were White and one was Black. During the discussion, one of the White students made offensive comments about affirmative action, and the researchers found that the observers looked significantly longer at the Black student when they believed he could hear the others' comments than when he could not.

"So how did they do the replication? With students at the University of Amsterdam!" Gilbert said. "They had Dutch students watch a video of Stanford students, speaking in English, about affirmative action policies at a university more than 5000 miles away."

In other words, unlike the participants in the original study, participants in the replication study watched students at a foreign university speaking in a foreign language about an issue of no relevance to them.

But according to Gilbert, that was not the most troubling part.

"If you dive deep into the data, you discover something else," Gilbert said. "The replicators realized that doing this study in the Netherlands might have been a problem, so they wisely decided to run another version of it in the US. And when they did, they basically replicated the original result. And yet, when the OSC estimated the reproducibility of psychological science, they excluded the successful replication and included only the one from the University of Amsterdam that failed. So the public hears that 'Yet another psychology study doesn't replicate' instead of 'Yet another psychology study replicates just fine if you do it right and not if you do it wrong' which isn't a very exciting headline. Some of the replications were quite faithful to the originals, but anyone who carefully reads all the replication reports will find many more examples like this one."

"These infidelities were a problem for another reason," King added, "namely, that they introduce additional error into the data set. That error can be calculated, and when we do, it turns out that the number of replication studies that actually failed is about what we should expect if every single one of the original findings had been true. Now, one could argue about how best to make this calculation, but the fact is that OSC didn't make it at all. They simply ignored this potent source of error, and that caused them to draw the wrong conclusions from their data. That doesn't mean that all one hundred studies were true, of course, but it does mean that this article provides no evidence to the contrary."

"So we now know that the infidelities created statistical noise," said Gilbert, "but was that all they did? Or were the infidelities of a certain kind? In other words, did they just tend to change the original result, or did they tend to change it in a particular way?"

"To find out," said King, "we needed a measure of how faithful each of the hundred replications was. Luckily, the OSC supplied it."

Before each replication began, the OSC asked the original authors to examine the planned replication study and say whether they would endorse it as a faithful replication of their work, and about 70 percent did so.

"We used this as a rough index of fidelity, and when we did, we discovered something important: The low-fidelity replications were an astonishing four times more likely to fail," King said. "What that suggests is that the infidelities did not just create random statistical noise--they actually biased the studies toward failure."

In their Technical Comment, Gilbert, King, Pettigrew, and Wilson also note that the OSC used a "low powered" design: They replicated each of the 100 studies once, using roughly the number of subjects that were used in the original studies. But according to King, this method artificially depresses the replication rate.

"To show how this happens, we took another published article that had examined the replicability of a group of classic psychology studies," said King. "The authors of that paper had used a very high-powered design--they replicated each study with more than thirty times the original number of participants--and that high-powered design produced a very high replication rate. So we asked a simple question: What would have happened if these authors had used the low-powered design that was used by the OSC? The answer is that the replication rate would have been even lower than the replication rate found by the OSC."

Despite uncovering serious problems with the landmark study, Gilbert and King emphasized that their critique does not suggest any wrongdoing and is simply part of the normal process of scientific inquiry.

"Let's be clear, Gilbert said. "No one involved in this study was trying to deceive anyone. They just made mistakes, as scientists sometimes do. Many of the OSC members are our friends, and the corresponding author, Brian Nosek, is actually a good friend who was both forthcoming and helpful to us as we wrote our critique," Gilbert said. "In fact, Brian is the one who suggested one of the methods we used for correcting the OSC's error calculations. So this is not a personal attack, this is a scientific critique. We all care about the same things: Doing science well and finding out what's true. We were glad to see that in their response to our comment, the OSC quibbled about a number of minor issues but conceded the major one, which is that their paper does not provide evidence for the pessimistic conclusions that most people have drawn from it."

"I think the big take-away point here is that meta-science must obey the rules of science," King said. "All the rules about sampling and calculating error and keeping experimenters blind to the hypothesis--all of those rules must apply whether you are studying people or studying the replicability of a science. Meta-science does not get a pass. It is not exempt. And those doing meta-science are not above the fray. They are part of the scientific process. If you violate the basic rules of science, you get the wrong answer, and that's what happened here."

"This paper has had extraordinary impact," Gilbert said. "It was Science magazine's number three 'Breakthrough of the Year' across all fields of science. It led to changes in policy at many scientific journals, changes in priorities at funding agencies, and it seriously undermined public perceptions of psychology. So it is not enough now, in the sober light of retrospect, to say that mistakes were made. These mistakes had very serious repercussions. We hope the OSC will now work as hard to correct the public misperceptions of their findings as they did to produce the findings themselves."

make a difference: sponsored opportunity

Story Source:

Materials provided by Harvard University. Note: Content may be edited for style and length.


Journal Reference:

  1. D. T. Gilbert, G. King, S. Pettigrew, T. D. Wilson. Comment on "Estimating the reproducibility of psychological science". Science, 2016; 351 (6277): 1037 DOI: 10.1126/science.aad7243

Cite This Page:

  • MLA
  • APA
  • Chicago
Harvard University. "Researchers overturn landmark study on the replicability of psychological science." ScienceDaily. ScienceDaily, 3 March 2016. <www.sciencedaily.com/releases/2016/03/160303145733.htm>.
Harvard University. (2016, March 3). Researchers overturn landmark study on the replicability of psychological science. ScienceDaily. Retrieved October 20, 2022 from www.sciencedaily.com/releases/2016/03/160303145733.htm
Harvard University. "Researchers overturn landmark study on the replicability of psychological science." ScienceDaily. www.sciencedaily.com/releases/2016/03/160303145733.htm (accessed October 20, 2022).

  • RELATED TOPICS
    • Mind & Brain
      • Numeracy
      • K-12 Education
      • Psychology
      • Child Psychology
    • Science & Society
      • STEM Education
      • Education and Employment
      • Educational Policy
      • Security and Defense
advertisement

  • RELATED TERMS
    • Psycholinguistics
    • Animal cognition
    • Cognitive neuroscience
    • Personality psychology
    • Psychology
    • Memory
    • Humanistic psychology
    • Mammal classification
advertisement

  Print   Email   Share

advertisement

1

2

3

4

5
Most Popular
this week

SCIENCE & SOCIETY
Did the Pandemic Change Our Personalities?
Watching TV With Your Child Can Help Their Cognitive Development, Study Suggests
Early Self-Regulation Boosts Children's Educational Success
BUSINESS & INDUSTRY
EDUCATION & LEARNING
Are Women Really Better at Finding and Remembering Words Than Men? Large Study Settles Score
Properties of 'Baby Talk' Similar Across Many Languages
advertisement

Strange & Offbeat
 

SCIENCE & SOCIETY
Scientists Develop Model That Adjusts Videogame Difficulty Based on Player Emotions
News Addiction Linked to Not Only Poor Mental Wellbeing but Physical Health Too, New Study Shows
Music-Making and the Flow of Aerosols
BUSINESS & INDUSTRY
People Prefer Interacting With Female Robots in Hotels, Study Finds
Securing Data Transfers With Relativity
Researchers Develop Artificial Intelligence That Can Detect Sarcasm in Social Media
EDUCATION & LEARNING
How the Brain Says 'Oops!'
Physics Race Pits Usain Bolt Against Jurassic Park Dinosaur
A Minecraft Build Can Be Used to Teach Almost Any Subject
Explore More
from ScienceDaily

RELATED STORIES

Exposure to Air Pollution Increases Violent Crime Rates
Oct. 3, 2019 — Breathing dirty air can make you sick. But according to new research, it can also make you more aggressive. That's the conclusion from a set of studies recently authored by Colorado State University ...
Major Study Finds One in Five Children Have Mental Health Problems
Apr. 17, 2019 — One in five Ontario children and youth suffer from a mental disorder, but less than one-third have had contact with a mental health care provider. A new study included 10,802 children and youth aged ...
Are Eyes the Window to Our Mistakes?
Mar. 11, 2019 — When humans make certain types of mistakes, the size of their pupils change. This may offer clues as to what's going on in the brain when people make suboptimal decisions, researchers have ...
Which Piece Resembles Your Color Perception for #theDress Image?
Mar. 23, 2018 — A novel algorithm to simulate the color appearance of objects under chromatic illuminants has been proposed. "#theDress image" refers to a photo that went viral on the Internet in February 2015, when ...
advertisement


SD
  • SD
    • Home Page
    • Top Science News
    • Latest News
  • Home
    • Home Page
    • Top Science News
    • Latest News
  • Health
    • View all the latest top news in the health sciences,
      or browse the topics below:
      Health & Medicine
      • Allergy
      • Alternative Medicine
      • Birth Control
      • Cancer
      • Diabetes
      • Diseases
      • Heart Disease
      • HIV and AIDS
      • Obesity
      • Stem Cells
      • ... more topics
      Mind & Brain
      • ADD and ADHD
      • Addiction
      • Alzheimer's
      • Autism
      • Depression
      • Headaches
      • Intelligence
      • Psychology
      • Relationships
      • Schizophrenia
      • ... more topics
      Living Well
      • Parenting
      • Pregnancy
      • Sexual Health
      • Skin Care
      • Men's Health
      • Women's Health
      • Nutrition
      • Diet and Weight Loss
      • Fitness
      • Healthy Aging
      • ... more topics
  • Tech
    • View all the latest top news in the physical sciences & technology,
      or browse the topics below:
      Matter & Energy
      • Aviation
      • Chemistry
      • Electronics
      • Fossil Fuels
      • Nanotechnology
      • Physics
      • Quantum Physics
      • Solar Energy
      • Technology
      • Wind Energy
      • ... more topics
      Space & Time
      • Astronomy
      • Black Holes
      • Dark Matter
      • Extrasolar Planets
      • Mars
      • Moon
      • Solar System
      • Space Telescopes
      • Stars
      • Sun
      • ... more topics
      Computers & Math
      • Artificial Intelligence
      • Communications
      • Computer Science
      • Hacking
      • Mathematics
      • Quantum Computers
      • Robotics
      • Software
      • Video Games
      • Virtual Reality
      • ... more topics
  • Enviro
    • View all the latest top news in the environmental sciences,
      or browse the topics below:
      Plants & Animals
      • Agriculture and Food
      • Animals
      • Biology
      • Biotechnology
      • Endangered Animals
      • Extinction
      • Genetically Modified
      • Microbes and More
      • New Species
      • Zoology
      • ... more topics
      Earth & Climate
      • Climate
      • Earthquakes
      • Environment
      • Geography
      • Geology
      • Global Warming
      • Hurricanes
      • Ozone Holes
      • Pollution
      • Weather
      • ... more topics
      Fossils & Ruins
      • Ancient Civilizations
      • Anthropology
      • Archaeology
      • Dinosaurs
      • Early Humans
      • Early Mammals
      • Evolution
      • Lost Treasures
      • Origin of Life
      • Paleontology
      • ... more topics
  • Society
    • View all the latest top news in the social sciences & education,
      or browse the topics below:
      Science & Society
      • Arts & Culture
      • Consumerism
      • Economics
      • Political Science
      • Privacy Issues
      • Public Health
      • Racial Disparity
      • Religion
      • Sports
      • World Development
      • ... more topics
      Business & Industry
      • Biotechnology & Bioengineering
      • Computers & Internet
      • Energy & Resources
      • Engineering
      • Medical Technology
      • Pharmaceuticals
      • Transportation
      • ... more topics
      Education & Learning
      • Animal Learning & Intelligence
      • Creativity
      • Educational Psychology
      • Educational Technology
      • Infant & Preschool Learning
      • Learning Disorders
      • STEM Education
      • ... more topics
  • Quirky
    • Top News
    • Human Quirks
    • Odd Creatures
    • Bizarre Things
    • Weird World
Free Subscriptions

Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

  • Email Newsletters
  • RSS Feeds
Follow Us

Keep up to date with the latest news from ScienceDaily via social networks:

  • Facebook
  • Twitter
  • LinkedIn
Have Feedback?

Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?

  • Leave Feedback
  • Contact Us
About This Site  |  Staff  |  Reviews  |  Contribute  |  Advertise  |  Privacy Policy  |  Editorial Policy  |  Terms of Use
Copyright 1995-2022 ScienceDaily or by other parties, where indicated. All rights controlled by their respective owners.
Content on this website is for information only. It is not intended to provide medical or other professional advice.
Views expressed here do not necessarily reflect those of ScienceDaily, its staff, its contributors, or its partners.
Financial support for ScienceDaily comes from advertisements and referral programs, where indicated.
— CCPA: Do Not Sell My Information — GDPR: Privacy Settings —