Litmus Review Theory of Change
How does reviewing papers on a website fix science? Here’s the plan.
Peer review is broken. The incentives in academic publishing all but guarantee that most published findings end up being false
The question is: who cares?
Not journalists, mostly. Not policymakers, mostly. But some people do need to actually know if a paper is good. For example:
- Patients picking treatments
- Parents looking into interventions for their kids
- Investors trying to figure out if a biotech result is going to replicate
These people have a specific question and they need the actual answer.
(By the way, shout out to all my nerds who are really trying their best to find the truth and write good papers for the love of the game. You’re the real MVPs!)
None of the groups who care are particularly powerful, but there are a lot of them, and when a study goes viral they’re already searching for whether it holds up. Right now they find Twitter fights. We’d like them to find a review with actual methodology behind it.
Nearby projects have been attempted, and some have seen some success, Consumer Reports is a closer match to Litmus Review. Manufacturers don’t always love being rated, but it turns out that consumers are keen for high quality, third party reviews. If enough individual people treat Litmus reviews as a useful check on whether a paper is solid, that creates a social signal. And social signals, once they’re strong enough, can start to change behavior further up the chain.
People who depend on a bad paper looking credible will be mad when it gets a bad rating, but they’re not the customer, and the rubric can weather the scrutiny. And if independent reviewers can apply the rubric and converge on similar ratings, then what we’ve built is a standard that can scale beyond us — the way Cochrane’s methods for systematic reviews became the standard even though Cochrane couldn’t review everything themselves.
There’s a timing argument too. LLM-generated papers are already showing up in the literature. The cost of producing something that looks like a plausible paper has dropped to almost nothing, and the institutions attempting quality control were already failing at the old volume. The volume of papers needing evaluation is about to hockey stick. That flood has to be met by a review system that operates at the same scale.
We’ve been building exactly that: a pipeline that extracts claims, assesses papers across ten methodological dimensions, and computes verdicts. A human still edits every review before it publishes. But the pipeline means we can process thousands of papers and put human attention where it matters most — the close calls and the things the automated assessment flags.
We built this pipeline because we wanted to get the evaluations right. It turns out it’s also infrastructure for handling the scale that’s coming.
So: build a review system that’s only beholden to the people who need a correct answer. Make the methodology transparent and mechanical enough to be a standard. Get useful enough that the social signal starts to matter. And be ready for the flood of LLM-generated papers with a pipeline that can meet it.