An older Claude pushes back on 95% of nonsense questions; newer top models on only 62–69%
BullshitBench asks 100 expert-sounding questions that make no sense. Newer top models play along more often than an older Claude did.
BullshitBench, built by Peter Gostev of Arena, checks whether a model notices that a question is broken. Its 100 questions mix jargon from software, finance, law, medicine and physics, using 13 different tricks, into questions that sound expert but mean nothing. A good answer says the question doesn’t make sense. A bad one answers it confidently.
On the October 2026 leaderboard, Claude Opus 4.8 with extended reasoning switched off pushed back clearly on 95% of questions. Newer flagship models did worse: Claude Opus 5.5 about 62–63% and GPT-6 Astra (max) 69%. In March 2026, when Arena first wrote it up, 88 model settings ranged from 2% to 91%, with Claude Sonnet 4.6 on top.
What it means
A model that confidently answers a question that makes no sense can mislead more than one that gets a hard question wrong. On this test, newer and more capable did not mean better at spotting nonsense, so a question built on a false premise may still get a confident answer. It is one narrow test, not a measure of overall accuracy.
Caveats
- It is one person’s question set. The questions are public, so later models may have seen them.
- Scores depend on settings such as reasoning effort, so the same model can rank very differently.
- Spotting nonsense is one narrow skill. It does not measure accuracy on real questions.