The Mark of a Criminal Record

The within-pair design for the criminal record question is strong. Random assignment of the conviction, weekly rotation between testers, matched resumes, and identical employer visits produce a clean test. Work histories were equalized so the criminal record tester reported prison work experience equivalent to the control tester's temp agency time. No differences were found by individual tester or by month.

The between-group race comparison is a different matter. The white and Black pairs applied to entirely separate pools of employers (150 for whites, 200 for Blacks), so the race comparison depends on those pools being equivalent. With only two testers per race, the study cannot separate racial discrimination from person-specific differences. The paper acknowledges this limitation. Most critically, in over 75% of applications, the tester had no substantive contact with anyone in a hiring role, and the paper does not describe how the decision-maker learned the applicant's race. Unlike Bertrand and Mullainathan (2004), which used racially distinctive names on written applications, this study describes no paper-based racial signal. The criminal record had a clear transmission channel — about three-quarters of employers asked about it directly on their application forms — but race had no equivalent.

The methods are correctly applied. The logistic regression uses clustered standard errors for paired observations, and the model controls for location and personal contact. The criminal record effect is large and precisely estimated. The race main effect is also clearly detected.

The problem is in how the paper handles results that did not clear its own statistical bar. The race-by-criminal-record interaction produced a coefficient smaller than its own margin of uncertainty. The data were equally consistent with no differential effect, or even one running in the opposite direction. The paper reports this honestly in a single sentence, then spends several pages interpreting the magnitude and direction of this non-result: calling the difference "nontrivial," calculating it as "40% larger for blacks than whites," and building the discussion around compounding disadvantage. You cannot accept a statistical framework when it validates your main effects and then set it aside when the interaction comes back empty.

The same pattern appears with the paper's most-cited single comparison — that white applicants with felony records (17% callbacks) fared better than Black applicants without records (14%). A footnote acknowledges this three-point difference is not distinguishable from zero. The main text presents it without qualification. This comparison also crosses both the between-group boundary and the experimental conditions, making it the weakest comparison the design permits.

The criminal record finding has strong construct validity. Within-pair randomization cleanly isolates the effect of the criminal record signal on employer behavior.

Every other major claim has a meaningful distance between what was measured and what is concluded. The paper frames its contribution as establishing the causal effect of incarceration on employment, but it tested something narrower: whether employers who learn about a conviction from an application form are less likely to call back. The study manipulated an information signal, not the experience of incarceration. It cannot speak to skill loss, network disruption, or institutional trauma.

The race findings suffer from the design asymmetry described above: the criminal record was conveyed through a direct question on 74% of application forms, but race had no verified transmission channel in the majority of cases. The "negative credential" concept — that the state's official certification gives the stigma its power — imposes a theoretical mechanism the data cannot distinguish from simpler alternatives like rational risk assessment.

The interaction claim is the most disconnected from the evidence. The study's own test could not tell whether the criminal record penalty differed by race. The data support a simpler story: being Black and having a felony conviction are two independent penalties that each cut callbacks roughly in half, stacking without amplification.

The criminal record result is clean and the reasoning around it is sound. The problem is that the paper is framed as a study of race, criminal records, and their intersection — and the reasoning used to support that larger frame is repeatedly undermined by the paper's own data.

The most consequential flaw: the paper lays out two competing predictions about the race-by-criminal-record interaction (records might hurt Black applicants less, or more), guaranteeing that whichever direction the data fall, a theoretical interpretation is ready. When the data showed no detectable interaction, the paper interpreted the direction of the non-result as supporting the compounding hypothesis. This is a framework that cannot lose.

The abstract states the study investigates "the consequences of incarceration for the employment outcomes of black and white job seekers" and concludes that "a criminal record presents a major barrier to employment, with important implications for racial disparities." The qualifying information — that the interaction is not distinguishable from chance, that the race comparison uses a weaker design, that the cross-race criminal record comparison is not distinguishable from zero — appears nowhere in the abstract. A reader of the abstract gets a materially different impression than a careful reader of the full paper.

The discussion claims the findings provide "conclusive evidence that mere contact with the criminal justice system severely limits subsequent employment opportunities." What the study actually tested — whether employers respond to criminal record information on application forms — is substantially narrower.

The paper discloses its funding sources clearly and includes an unusually detailed methodological appendix for its era. It discusses generalizability limits: college students as testers, restriction to one city, exclusion of legally barred occupations. It acknowledges that the between-group race comparison cannot rule out unmeasured tester differences.

No data, analysis code, application materials, or pre-registration are available — not unusual for 2003, but limiting for verification. No ethics review is discussed, despite employer deception. The limitations discussion is selective: the paper is transparent about problems that do not threaten its conclusions (tester representativeness, geographic scope) and silent about the ones that do (how employers learned applicants' race in no-contact applications, and the implications of building headline findings on statistically non-significant results).

The citation practices are strong for the criminal record finding. The paper honestly presents studies arguing incarceration has little independent employment effect (Kling 1999, Grogger 1995, Needels 1996) alongside those showing strong effects, and positions the experiment as resolving this disagreement.

For the race claims, the citations become more selective. The paper compares its race effect to findings from Bendick, Jackson, and Reinoso (1994) and Bertrand and Mullainathan (2004) — but those studies used cross-race matched pairs at the same employer, a stronger design. Presenting the results as equivalent implicitly borrows credibility from the better designs. The stereotype literature cited to motivate the interaction hypothesis provides theoretical reasons to expect compounding, but cannot substitute for evidence. Heckman and Siegelman (1993), whose concerns about unmeasured tester differences are most relevant to the between-group comparison, are acknowledged in a footnote but not seriously engaged.

Two of the four funders — the Joyce Foundation and the Soros Foundation (now Open Society Foundations) — have longstanding commitments to criminal justice reform. The study's policy conclusions — recommending alternatives to incarceration and restricting employer access to criminal records — align with those missions. The other two (NSF and NIJ) are government agencies with no directional stake.

The core criminal record finding is methodologically sound and would likely emerge regardless of funding source — the within-pair randomization could have shown employers don't care about criminal records, and the design would have detected that. The alignment concern is subtler: the paper's framing consistently elevates its weakest findings into headline results, in a direction that serves both the funders' advocacy goals and the career incentives of a doctoral dissertation.