False-Positive Psychology

The simulations are appropriate for the question: if you want to know what happens to error rates when researchers make flexible choices, you simulate it under conditions where you know the truth and count the mistakes. The demonstration experiments are the strongest element — choosing to "prove" that a song changes your birth date is inspired, because any positive result is guaranteed to be a false alarm. The proposed solution rests entirely on theoretical argument and a single worked example showing that the disclosure rules would have exposed the manipulation in the Beatles study. Whether the rules would work across diverse research contexts, whether reviewers would enforce them, and whether the burden is truly "minimal" are all untested.

The individual simulations are well-specified and produce results consistent with established theory. With two independent tests at a 5% threshold, probability theory predicts a false-positive rate of about 9.75%; the reported 9.5% for two correlated outcome measures is slightly lower, as expected. The sequential-testing result — a 22% false-positive rate when checking for significance after every new observation — matches well-known results from the statistical literature.

The combined scenario that produces the headline 61% figure is less transparent. The paper does not enumerate how many tests are run when all four forms of flexibility are combined; working through the possibilities suggests at least 72 tests per simulated dataset, but the exact count is unstated. The simulation code was not published — an irony for a paper whose central message is that transparency prevents false positives. A reader who wanted to independently verify the 61% figure could not do so from the paper alone.

The paper also presents worst-case results in language that implies they describe typical research. The sentence "a researcher is more likely than not to falsely detect a significant effect" is true only for a researcher who simultaneously exploits all four forms of flexibility.

The core argument is mathematically airtight: if you test multiple analyses on the same data and report whichever one crosses the threshold, the chance of a false positive is necessarily higher than 5%. The simulations quantify how much higher for specific scenarios, and the Beatles demonstration proves it can happen with real data and a false hypothesis.

The argument weakens at two points. First, the bridge from simulation to practice requires knowing how often researchers jointly exploit multiple forms of flexibility, and the paper provides no data on this — only survey evidence that individual practices are common. The 61% figure is a ceiling, but the paper sometimes writes as if it were a base rate. Second, the claim that disclosure would "substantially mitigate" the problem rests on a single worked example plus the theoretical argument that mandatory disclosure turns omissions into verifiable false statements. Whether violations would be detected, whether institutions would impose consequences, and whether early adoption would spread are predictions about institutional behavior presented as logical consequences of the disclosure requirement itself.

The concept of "researcher degrees of freedom" — the many post-hoc analytic choices that can steer results — does real intellectual work. It carves a genuine distinction between making an analytic choice before seeing how it affects the result versus making it after.

The framework is sharpest where it is most concrete — the individual simulations, the demonstration experiments, the specific disclosure requirements — and becomes imprecise at its most consequential claims. The paper provides no data on how often researchers jointly exploit multiple forms of flexibility, so the bridge from "what happens under maximal exploitation" to "what happens in actual research" is unclear. The claim that disclosure "substantially mitigates" the problem lacks any operational definition of "substantially." The motivated-reasoning account — that researchers deceive themselves rather than knowingly gaming the system — is plausible but assumed by analogy rather than tested; a researcher who strategically exploits flexibility would produce identical behavior, and the paper provides no way to distinguish the two.

The simulations and demonstration show exactly what they claim to show, and their conclusions follow directly from their setups. The framing problem emerges when the paper moves to broader claims. The abstract states that "in many cases, a researcher is more likely than not to falsely find evidence that an effect exists" — but the only scenario producing rates above 50% combines all four forms of flexibility.

The proposed solution isn't a solid as the demonstration of the problem. The paper leads with the absurd demonstration, builds alarm through the simulations, and then presents the disclosure requirements as "straightforwardly effective" without subjecting them to anything like the scrutiny applied to the problem. Whether the six requirements cover the most important forms of flexibility, whether reviewers would enforce them, and whether researchers would comply honestly are not examined.

The paper's premises are mostly explicit and its simulation scenarios are clearly described. The self-disclosure in the Beatles experiment is an act of genuine transparency.

There are two main problems. First, the simulation code for the combined scenario was not published, and the paper does not state how many tests were run to produce the 61% figure. Second, the proposed fix is described as imposing 'minimal burden' and providing 'substantial' mitigation, but neither claim is defined or measured.

The paper does not distinguish clearly between what it has demonstrated and what it has merely argued — the simulations and the Beatles experiment are demonstrations, while the proposed solution is an untested recommendation presented with the confidence of an established result. The paper also does not engage with pre-registration as an alternative, despite pre-registration targeting the exact mechanism the paper identifies (undisclosed post-hoc analytic choices) by eliminating the flexibility rather than merely disclosing it.

The paper's reference list is short but appropriate for a paper whose central evidence is self-generated. The motivated-reasoning citations (Kunda 1990, Babcock and Loewenstein 1997, and others) are used fairly to establish psychological plausibility, not to carry empirical weight. The one essential external citation — John, Loewenstein, and Prelec 2011, a survey finding that roughly 70% of behavioral scientists admitted to optional stopping — was unpublished at the time but has since been confirmed.

The dismissal of Bayesian methods is underdeveloped. The paper cites only Wagenmakers and colleagues (2011) and argues that Bayesian statistics add degrees of freedom, but does not engage with the strongest Bayesian counterargument: that under Bayesian inference, optional stopping is theoretically less problematic because conclusions don't depend on the researcher's data-collection intentions. This leaves the "nonsolutions" section weaker than it could be, though it is peripheral to the paper's core contribution.

The authors — academic psychologists at Penn and Berkeley — declared no conflicts of interest and disclosed no funding. There is no product under evaluation and no industry party whose interests align with any outcome. The paper's conclusions cut against the authors' own professional community, and they explicitly include themselves in the criticism.