Estimating the reproducibility of psychological science

The protocol design is exemplary: pre-registration, original-author consultation, independent re-analysis, and full public archiving represent a level of procedural rigor rarely achieved at this scale. For the descriptive question — how often do published results hold up? — the design works well.

The comparison between original and replication success rates is structurally confounded by publication bias in the originals. Studies that barely crossed the threshold for publication are expected to produce weaker results on the next attempt purely from regression to the mean, even when the underlying effect is real. The study cannot separate this expected regression from genuine irreproducibility.

Teams self-selected which studies to replicate, and resource-intensive studies were systematically underrepresented. Each multi-study article (76 of 100 had multiple experiments) was reduced to a single statistical test, typically from the last experiment — which may not represent the article's central contribution. The originals were published in 2008 and replications ran in 2012–2014, introducing a four-to-six-year temporal and cultural shift that matters especially for context-dependent social psychology effects. The paper acknowledges most of these limitations, but its headline framing does not reflect them.

The descriptive statistics are transparently computed, independently verified, and publicly reproducible. All effects were converted to a common scale, confidence intervals are reported, and the analysis code is publicly available.

The most consequential issue is the power calculations. The paper reports that replications had high power (averaging 92%) to detect the original effect sizes. But the paper's own central finding is that original effect sizes were roughly double the replication effect sizes. If the true effect is half the published size, the sample needed to reliably detect it roughly quadruples. Many replications that appear well-powered were likely too small to detect the actual effects. The claim that findings weakened "despite high power" is internally inconsistent with the paper's own evidence.

The moderator analysis correlates 13 predictor variables against 5 outcomes, producing 65 correlations with no correction for multiple testing. At that scale, roughly three false positives are expected by chance. The claim that "surprisingness" predicted replication failure rests on modest correlations that might not survive correction. The comparison between evidence-strength predictors and team-quality predictors is structurally unfair: the evidence-strength variables are precise numbers from publications, while team quality is measured through noisy self-reports — and teams were deliberately matched to studies by expertise, compressing the range of variation. No formal test was conducted to determine whether the difference in predictive power was itself meaningful.

The descriptive measurements are clean: the study computed five different indicators of replication success and reported all of them. But their claims outstrip their measurements.

Every core claim stretches beyond what the design supports. The 36%-versus-97% comparison treats the original success rate as a neutral benchmark when it is inflated by publication bias. The "halving" of effect sizes conflates genuine replication failure with the expected correction of inflated baselines. The claim that findings weakened "despite original materials, pre-reviewed methods, and high statistical power" packages three qualifiers that are each less solid than they sound: the fraction of replications that actually received original materials is never reported; the power calculations assumed the original (inflated) effect sizes were correct; and protocol review did not verify execution. The claim that original evidence strength predicted success better than team quality rests on a comparison the precisely measured variable will almost always win, regardless of which actually matters.

The paper's title refers to "psychological science" broadly, but the evidence comes from 100 studies in three journals in one year, covering only social and cognitive psychology, with resource-intensive studies excluded.

The paper is more restrained than much of the commentary it generated. The discussion uses appropriate hedging, and the conclusion explicitly states that the project established zero effects as definitively true and zero as definitively false — an unusually honest framing for a high-profile paper.

The problems are in the interpretive packaging that precedes those caveats. The abstract leads with the most dramatic number (36%) without noting that some decline from 97% is guaranteed by regression to the mean. The "despite high power" qualifier does much of the rhetorical work in the central conclusion, yet it is internally inconsistent with the paper's evidence. The paper never estimates how much of the observed decline is expected on purely statistical grounds. Three alternative explanations are insufficiently addressed: regression to the mean, temporal and contextual shifts between 2008 and 2012–2014, and the much higher power requirements for detecting interaction effects (which replicated at less than half the rate of simpler effects).

This is one of the most transparent large-scale studies in the social sciences. Every replication was pre-registered. All data, materials, and analysis code are publicly archived. Every replication analysis was independently reproduced by a separate analyst. The limitations section is substantive and candid.

Minor gaps: the paper does not mention ethics board approval for the 100 replications involving human participants; the fraction of replications that actually received and used original study materials is never quantified, despite this being a key qualifier; and the true statistical power, given inflated original effect sizes, is not discussed.

The core findings rest entirely on original, publicly available data rather than on cited claims, which limits the scope for citation problems. The contextual citations are accurately represented — the paper adds an honest caveat to the Begley and Ellis (2012) finding about cancer biology replication rates, noting that those numbers are "difficult to interpret because no details are available." The reference list draws on peer-reviewed work across several decades, and self-citations are relevant. The main omission is that the Many Labs project (Klein 2014), which found that 10 of 13 tested effects replicated across dozens of labs, is cited as a complementary approach rather than engaged as context for why replication rates vary depending on which effects are tested.

The corresponding author co-founded the Center for Open Science (COS), which funded this study alongside the Laura and John Arnold Foundation (now Arnold Ventures). Both organizations have institutional interests served by findings of low reproducibility — a study showing that most psychology findings replicate cleanly would have weakened the case for their continued work and funding.

The study includes strong safeguards: pre-registration, fully open data, and independent re-analysis make it possible for anyone to verify the underlying numbers. But framing choices consistently serve the funder's narrative. The most dramatic of the five replication indicators (36%) became the headline number. The conclusion transitions into advocacy for the Transparency and Openness Promotion Guidelines, a COS initiative. These are prosocial interests, not commercial ones, and the data are publicly verifiable — but institutional alignment between funder mission and paper framing is present.