AI
How to read an AI benchmark without being fooled
Every model launch arrives with a bar chart on which, by remarkable coincidence, the new model wins. Here's the small print they're hoping you won't read — and a five-question checklist to run before believing any of it.
There's a genre of chart you've seen a hundred times by now: five coloured bars, one triumphant, footnotes in six-point type. AI benchmarks are how the industry keeps score, and as of early 2026 they've also become how the industry markets — which means reading them now requires the same scepticism you'd bring to a mattress advert claiming to be voted best mattress. By the mattress.
What a benchmark actually is
A benchmark is a fixed set of questions with agreed marking: thousands of multiple-choice academic questions (the MMLU family), competition maths problems, PhD-level science questions (GPQA), or real software bugs to fix (SWE-bench). Run the model, count the passes, publish the percentage. The idea is sound — standardised exams beat anecdotes — and the good ones are genuinely informative. The trouble starts because a model's exam grade, like a teenager's, is an imperfect proxy for how it behaves once out in the world.
The six tricks to check for
1. The menu was chosen after the meal. There are dozens of respectable benchmarks. A launch deck shows you six. Guess how those six were selected. When a vendor's chart omits a benchmark everyone normally quotes, that omission is data.
2. The conditions don't match. Scores swing wildly with how the exam is sat: how many attempts count (pass@1 versus best-of-several), whether the model gets tools, examples, or a large "thinking" budget before answering. Comparing your model's best-dressed score against a rival's default score is the industry's favourite sleight of hand — always read the footnote that starts "evaluated with".
3. The model may have seen the paper. Benchmarks published on the internet have a way of ending up in training data. This is contamination, and it inflates scores in ways even honest labs struggle to fully rule out. It's why newer, held-back test sets and regularly refreshed exams get more respect.
4. The exam is full. Once every serious model scores 90-something on a benchmark, differences shrink to noise. A two-point win on a saturated exam is a rounding error wearing a press release.
5. There are no error bars. Re-run the same model on the same benchmark and scores wobble a point or two. Yet launch charts present single numbers with the confidence of a physics constant. Any gap smaller than the wobble is decoration.
6. The exam isn't your job. The deepest problem: a model brilliant at competition maths can still mangle your invoice summaries. Benchmark rank and usefulness-for-your-task correlate — loosely. This is precisely the gap that matters when buying agent software for a small business, where multi-step reliability, not exam brilliance, decides everything.
A two-point win on a saturated exam is a rounding error wearing a press release.
What to trust instead
Three habits serve better than launch-day charts. Watch independent aggregators — community leaderboards, head-to-head preference arenas and third-party evaluation outfits — which are imperfect but at least not marking their own homework. Watch trajectories rather than snapshots: a model family improving steadily across several evaluations means more than one spike. And watch the deployment signals — pricing, usage caps and which model a vendor routes its own paying traffic to reveal what the lab really thinks of its models, a tell we also lean on in our AI pricing analysis. The same literacy applies down the stack, incidentally: hardware vendors play matching games with inference speed claims, as we noted in our inference-chips explainer.
The five-question checklist
Before repeating any benchmark claim, ask: Who ran the evaluation — the vendor or a third party? Were all models tested under the same conditions, attempts and tool access? Is the benchmark recent enough to resist contamination and not yet saturated? Is the winning margin bigger than run-to-run noise? And does the tested skill resemble anything you'd actually use the model for? Five noes means you're reading an advert.
None of this is a counsel of despair. Benchmarks remain the least-bad way to track a fast-moving field, and used carefully — plural, independent, conditions-matched — they'll tell you roughly where the frontier is. Just remember the golden rule, as of early 2026 and probably forever: the best benchmark is your own workload, run twice, once on each model. Everything else is a bar chart with a marketing budget.