Science Doesn't Have to Be Bullshit
You can’t tell from an abstract whether a paper’s finding is real. Neither can the journalist who wrote the article about it, or the person who sent you the article.
Whether this is a serious problem is contested, but the pessimistic case is serious: Ioannidis 2005
Good news though: Protzko, 2023, Nature Human Behaviour showed that preregistered, well-powered studies replicate at around 86%—just kidding! That paper was retracted by Nature for fraud. Note that more than one of the studies above that’s trying to establish how dubious the practice of science has become, is itself dubious! Errington in particular is fun because 74% of the preclinical cancer studies it tried to include were so poorly documented, replication could not even be attempted.
I’m not going to relitigate this here—I think there’s a problem.
Every day I see clickbait news articles about nonsense papers, and I know not to update my beliefs in any way about them. How do I know? Some claims are obviously impossible to even measure, or the brief description of the methods preclude any real evidence having been gathered. Over the years I’ve spot checked many and found that the world shattering title is supported by a handful of psychology undergrads answering some leading questions on a survey. Or I’ve had a serious question and looked into the literature for an answer only to find a pyramid scheme of exaggerations, nonsense, and bullshit
Stuff that peer review is supposed to have stopped before I saw it. Obviously that’s not happening.
If only someone was somehow far enough outside of the incentive structures to be unaffected, but close enough to the math and science to evaluate it.
Litmus Review
We review science literature and rate it. That’s the whole project.
Independent. No money from institutions, publishers, or anyone with a stake in a particular verdict. No grant to renew, no relationship to protect.
No agenda beyond method. We don’t care what a paper concludes. We care whether the design could have detected the opposite.
Equal-opportunity grumps. We’re not out to discredit a field, a school of thought, or a side. The rubric gets applied to everything the same way.
Transparent. The rubric is published. Each criterion is scored separately against stated reading procedures and combined mechanically. You can disagree with a rating and see exactly where it came from. Arguments about our verdicts should land on the rubric, in public, where they can be settled.
Anonymous for now. We haven’t exactly tried hard to stay anonymous — you could probably figure it out in ten minutes and you’d be underwhelmed. Are we blue-haired commies? MAGA chuds? Basement dwellers? Department chairs? One of the pillars of science is that the work speaks for itself. The rubric is public, the reasoning is shown, the corrections log is open. We’re no one you’ve heard of, probably, and we have no particular axe to grind except with the institution of Science itself. Conflicts of interest are real, of course (they are part of our rubric!), so in the fullness of time the big-deal institution we grow into will obviously need to be transparent about its staff and officers and entanglements. But while we’re still just a scrappy website, I think we’re best off being a boring, little royal “we.”
Approach
Just to be cute and poetic, our first reviews were Frankfurt 1986, On Bullshit
The general method selection is slightly more rigorous: we crawl the citations graph of new and popular papers, and look for papers that are widely cited, and hopefully central to the field.
-
We’ve scanned the citations of roughly 40,000 papers so far and ranked them by how “load bearing” they are—I swear, that’s the real word we real human meat people used to mark citations that were important to a paper instead of incidental or peripheral. The kind that if they turned out to be wrong would also compromise the findings of the paper that cited it. We’re working down that list.
-
We’re also deliberately reviewing papers that have been through replication projects — studies where someone already ran the experiment again and we know whether the result held up. We rate the paper blind, then check. If our ratings predict which papers replicate, the rubric is measuring something real. If they don’t, we have work to do.
The dream is that “it’s been proven that X” becomes something you can look up.
The prediction
Papers get one of four ratings, based on a handful of subscores: nonsense, dubious, credible, solid. We’ve published fewer than ten reviews so far. Before the numbers accumulate, here’s our prediction on the record.
Across our first 200 published reviews:
| Rating | Predicted share |
|---|---|
| Nonsense | 20% |
| Dubious | 45% |
| Credible | 30% |
| Solid | 5% |
We’re publishing this now so that when the distribution comes out looking harsh, everyone knows we could not have tuned the rubric to produce it.
Where the numbers come from
They’re anchored on replication rates in, for example, the literature above. Replication failure and bad methodology share root causes — like p-hacking, low power, publication bias — so replication rates are the best available proxy for how much of the literature holds up under scrutiny. They’re not the same question: a methodologically solid paper can fail to replicate by chance, and a sloppy one can replicate if the effect is large enough. But the overlap is enough to anchor on.
Our corpus pushes the numbers further. Load-bearing papers aren’t better than average, and some are prominent precisely because they reported something surprising, which correlates with being wrong.
How to check us
The review rubric is public and versioned, and every rating shows the version that produced it. If independent reviewers apply the published rubric to the same paper and diverge by more than one level, the rubric is underspecified — we want to know.
We will keep a public corrections log. Where a review misread what a paper did — wrong sample size, missed preregistration, misattributed analysis — authors can contest that specific input, and upheld corrections stay logged with the original error visible.
If the ratings come out as predicted, draw your own conclusions. That’s rather the point.