Investigating the replicability of preclinical cancer biology

The basic design is well-suited to its question: independent labs repeated experiments under preregistered, peer-reviewed protocols, and results were reported regardless of outcome. Using seven criteria for replication success rather than a single pass/fail verdict is sensible given the lack of consensus on defining replication. Sample sizes were planned to detect the original effect size.

The 74% experiment dropout rate is the most damaging limitation. Only 50 of 193 planned experiments could be completed, and the reasons for non-completion (poor documentation, withheld materials) could plausibly correlate with replicability. Whether this biases the completed sample toward easier or harder replications is unknown. The sample of 23 papers from top-tier journals in a three-year window is small and deliberately non-representative. The study also cannot separate expected statistical shrinkage from regression to the mean from genuine failures — and does not attempt to model how much shrinkage to expect from selection effects alone. For animal experiments, environmental factors (housing, microbiome, handling) differ across labs and cannot be transmitted in a protocol. None of the original animal experiments reported blinding, and only one reported randomization — informative about rigor, though this may partly reflect reporting norms rather than actual practice.

The statistical methods are well-chosen and transparently reported. Spearman rank correlation for skewed effect sizes, harmonic mean methods for correlated nested data, robust standard errors for multi-level models, and fixed-effect meta-analysis for two-study pairs are all appropriate and clearly justified. Effect sizes are reported throughout with means, medians, confidence intervals, and sensitivity analyses.

The main concerns are interpretive. Replications were powered to detect the original effect sizes, but those originals were dramatically inflated. If true effects are much smaller, the replications were substantially underpowered, depressing the 43% replication rate. The paper does not model what the expected rate would be under realistic true effect sizes. The moderator analysis — testing whether five characteristics predicted replication success — had far too few observations (97, with five correlated predictors) to detect real effects; the null finding means "the study could not tell," not "these factors don't matter." The comparison between null and positive effect replication rates rests on only 15 null effects — changing three outcomes would shift the null rate from 80% to 60%. Some headline statistics (like the probability of 94 of 97 effects shrinking) treat nested effects as independent, making the stated probabilities more extreme than they should be, though the qualitative conclusion would survive correction.

The finding that 94 of 97 replications of positive effects produced smaller results is real and not a fluke.

The problem is what these measurements mean. The 85% median shrinkage figure conflates genuine non-replicability with expected statistical regression from selecting high-impact results, and the paper never models how much of each is at work. The 46% "overall success rate" combines five partially redundant criteria (correlations up to 0.78) and mixes two different populations — 97 positive effects (40% success) and 15 null effects (80% success) — obscuring more than it reveals. The comparison between null and positive replication rates rests on far too few null effects to be reliable. The broadest claim — that this reveals something about "preclinical cancer biology" as a field — stretches a sample of 23 papers from elite journals, filtered to the 26% of experiments that could be completed, into a generalization the data cannot support.

Two claims in the discussion are disconnected from the data entirely. The attribution of low replication to "dysfunctional incentives" draws on external literature, not on anything this study measured — the data are equally consistent with biological complexity, tacit knowledge, or regression to the mean. The claim that results "reinforce concerns about the basis for launching clinical trials" involves no clinical trial data. The one claim that holds up cleanly is that original effect sizes are systematically overestimated in this sample — only 3 of 97 replications were larger than the original, a pattern too extreme to be chance.

The paper's empirical reporting is careful and transparent. Presenting results under seven criteria, at four levels of aggregation, and making all data public allows readers to draw their own conclusions. The paper acknowledges many of its limitations forthrightly.

The reasoning weakens in interpretation. The headline finding — 85% effect size shrinkage — sounds devastating, but the paper never quantifies how much shrinkage you'd expect from re-testing results selected for being large. Without that benchmark, the reader cannot assess the severity of what was found. The paper's one formal benchmark (comparing the observed 43% replication rate to an expected 87% rate) is informative, but the 85% shrinkage figure — which dominates public discussion — has no equivalent baseline.

The causal narrative is the weakest link. The paper attributes low replication to "dysfunctional incentives" and suggests clinical trials may rest on weak evidence, but nothing in the data tests these claims. These are editorial positions supported by citations to other people's arguments, presented alongside the study's own findings in a way that makes them easy to mistake for evidence-based conclusions. The 74% experiment non-completion rate is treated primarily as a finding about research culture rather than as a potentially fatal threat to the paper's own replication-rate estimates — a framing choice that minimizes its implications for the headline numbers.

This is one of the most transparent large-scale studies in biomedical research. Every replication protocol was preregistered and peer-reviewed before data collection. All data, code, and materials are publicly available with data dictionaries. The decision letter and author response are published. Conflicts of interest — including the institutional interests of the Center for Open Science and Science Exchange — are disclosed in detail. The meta-analytic methods are acknowledged as post hoc. The limitations section directly confronts the study's most serious vulnerabilities, including the 74% attrition rate and the unknown direction of the resulting selection bias. This level of openness is exemplary.

The paper's empirical claims rest on its own data and do not depend on external sources. The citations motivating the project — including the Amgen report (Begley and Ellis 2012)Begley and Ellis 2012) finding only 11% confirmation of promising findings, and the Bayer report (Prinz 2011) at 25% — are accurately characterized, and the paper correctly notes their lack of methodological transparency. Citations supporting the "dysfunctional incentives" narrative (Ioannidis 2005, Simmons 2011, Greenwald 1975) are generally accurate, though Ioannidis's conditional argument that most findings may be false is slightly simplified into a blanket claim.

Two citation weaknesses stand out. First, work questioning whether the "crisis" framing is overstated is absent — the interpretive scaffold draws entirely from one intellectual tradition, and several cited papers are by co-authors of this study (particularly Brian Nosek). Second, the substantial literature on biology-specific sources of variability — cell line drift, microbiome differences across facilities, reagent batch effects — receives minimal attention, despite being directly relevant to interpreting replication failures in this domain.

The structural conflicts are unusually direct. Five of seven authors were employed by the Center for Open Science, whose mission and funding depend on the existence of a reproducibility problem. Two authors were employed by and held equity in Science Exchange, which profits from outsourced replication services. The funder, Arnold Ventures, has a well-known portfolio interest in research integrity reform. The entire project team shared an institutional stake in findings that demonstrate low replicability.

These conflicts are substantially mitigated by design choices that constrain bias: preregistered protocols, peer review before data collection, complete reporting, public data and code, and the paper's presentation of findings showing real signal in the original literature (79% same-direction effects, strong rank correlation between original and replication sizes). An advocacy document would be less likely to highlight these. Still, the conclusions — and the reform prescriptions they support — align precisely with the interests of the organizations that produced the study.