Evaluating replicability of laboratory experiments in economics

The core design fits its primary question. Pre-registered analysis plans, replications powered above 90%, and multiple replication-success measures represent strong methodological choices. But the design cannot support the paper's secondary claims. The 18 replications were run at six labs across four countries with different subject pools, languages, and cultural norms. The paper states that protocols were sent to original authors "for verification" but never reports whether authors responded, flagged problems, or whether manipulation checks were performed. Any replication failure could reflect a genuine false original, a subtle protocol deviation, or a cultural mismatch — the study cannot distinguish these. The cross-discipline comparison with psychology is confounded by differences in journals sampled, time periods, finding selection methods, design types, and statistical power. The comparison to macroeconomic replication studies conflates two different activities: running a new experiment versus re-running someone's analysis code on their own data. With only 18 studies, subgroup analyses are essentially anecdotal.

The nonparametric tests — paired comparisons of effect sizes and rank correlations — are a good fit for a study with only 18 observations, where stronger statistical assumptions would be hard to justify. Confidence intervals are reported throughout, and the wide uncertainty around the 61% rate (roughly 36% to 86%) is not hidden. The central interpretive problem is the binomial test comparing 61% observed to 92% expected. The 92% benchmark assumes original effect sizes are accurate, but the paper's own finding that replicated effects average only two-thirds of the originals directly contradicts this. Recalculating expected replication rates using the actual (smaller) effects would narrow the difference substantially, perhaps to insignificance. Twelve correlations between predictors and replication outcomes are reported without adjustment for multiple comparisons, and with 18 observations, removing a single study could substantially change the results. The meta-analytic estimates combine original and replication data without specifying whether the analysis used fixed or random effects — a choice that matters when the studies have different true effect sizes.

The descriptive claims are well-measured. Counting how many of 18 replications crossed a conventional threshold is straightforward, and the average effect-size ratio is clean arithmetic. But two of the paper's most prominent claims are disconnected from their evidence. The claim that the replication rate is "considerably lower" than expected rests on the 92% benchmark — built from original effect sizes the paper itself shows are inflated by roughly a third. The dramatic-sounding difference is substantially an artifact of an inflated comparison point. The claim that economics experiments replicate better than psychology experiments treats "discipline" as the explanatory variable, but the two replication projects differ in finding selection (most important versus random), design types, statistical power, journals, time periods, and sample sizes. No inference about disciplinary differences is possible from this comparison. The claim that studies with stronger original evidence predict replication success is close to tautological — it restates the truism that bigger effects are easier to detect again. The comparison to macroeconomic replication rates (Dewald 1986, McCullough 2006) compares new-data replication to same-data computational reproduction — categorically different activities that cannot be meaningfully compared.

The paper's interpretive framing consistently outruns its evidence. The 92% benchmark is featured prominently throughout, despite the paper's data showing its premise is wrong. The comparison to psychology receives a dedicated figure and extended discussion, with a single-sentence caveat ("it is premature to draw strong conclusions") that does not match the weight the rest of the paper places on it. The speculation that economics' norms — financial incentives, non-deception policies, transparency — explain its replication advantage is offered without any data connecting these norms to outcomes. Equally plausible alternatives (simpler experimental designs, stronger journal gatekeeping, or the mechanical effect of selecting the "most important" finding) receive no discussion. The paper does not mention regression to the mean as a partial explanation for effect-size shrinkage, even though studies selected for having cleared a significance threshold will naturally produce smaller effects on replication through ordinary sampling variability. The abstract presents replication rates of 67–78% for additional indicators without noting that the higher figures include borderline cases and assume no publication bias in the originals — caveats that appear only in the body text.

All 18 analysis plans were pre-registered and made public before data collection. Replication data, analysis code, and full replication reports are posted at the Open Science Framework. Funding is disclosed in detail, and the authors state they have no conflicts of interest. The one meaningful omission: the paper never reports whether original authors actually confirmed the replication protocols were accurate, how many responded, or whether corrections were made. For a study measuring replication rates, unverified protocol fidelity is a real limitation, though it does not affect the transparency of the analysis itself.

The paper's primary citations are accurate and its empirical data well-documented. The problems arise with the two most prominent comparative claims. The comparison to the [Reproducibility Project: Psychology (Open Science Collaboration 2015)]](reviews/papers/open-science-collaboration-2015-estimating-the-reproducibility-of-psychological) is cited fairly — the 36% rate is correctly reported — but the interpretive weight placed on it far exceeds what an 18-study sample can support, given at least six confounding methodological differences. More seriously, the citations used to argue that lab experiments replicate better than other empirical economics (Dewald 1986, McCullough 2006) are mischaracterized. Those studies measured whether results could be regenerated from the same data and code — not whether effects appeared in new data. Framing a 61% new-data replication rate alongside 13–23% same-data reproduction rates implies a comparison that is not meaningful. The paper also does not engage with the Many Labs projects, which found that well-known psychology effects replicated at higher rates than the Reproducibility Project average — a finding that would complicate the disciplinary comparison.

There are no financial conflicts — all funding comes from public and academic sources. The relevant pressure is reputational. Several lead authors are prominent figures in the replication reform movement and are also experimental economists, giving them dual incentives: demonstrating that replication failures are real (validating reform) while showing that economics performs well relative to other fields (validating their discipline). Design choices align with these interests: only the two most prestigious economics journals were sampled, only between-subject designs were included, and the "most important" finding was selected from each paper — collectively constructing the most favorable possible sample for a replication test. The paper does not frame it this way. Pre-registration and open data guard against selective reporting of results but not against selective framing.