Somewhere along the way, the AI industry decided that the way to prove a model is good is to show it scoring well on a standardized test. MMLU, HumanEval, MATH, GPQA - a rotating cast of benchmarks that the major labs treat as the primary signal of progress. The problem isn’t that benchmarks exist. It’s that they’ve become the product.
The Goodhart Trap, Running at Scale
Goodhart’s Law - when a measure becomes a target, it ceases to be a good measure - has always applied to machine learning. But with the current generation of frontier models, the contamination problem is severe enough that benchmark scores are nearly meaningless as a proxy for real capability. Training data that overlaps with evaluation sets isn’t some edge-case failure. It’s structurally hard to avoid when you’re scraping most of the legible internet, which has been documenting these benchmarks and their answers for years. Labs like OpenAI, Google DeepMind, and Anthropic all publish caveats about this. The caveats don’t make it into the headlines.
What does make it into the headlines is “Model X achieves state-of-the-art on [benchmark].” That framing shapes investor expectations, press coverage, and - consequentially - the internal incentives of the teams building these models.
What Optimizing for Evals Actually Produces
A model that aces MATH at a PhD level can still give a user confidently wrong instructions for filing an amended tax return. A model that tops HumanEval can still produce code that compiles cleanly and does the wrong thing. These aren’t corner cases; they’re consistent failure modes that practitioners encounter constantly. The benchmark measures the performance envelope on a specific task distribution. Real-world queries come from a different distribution entirely.
The labs know this. Some of them are investing in more naturalistic evaluation methods - red-teaming, user studies, deployment monitoring. But that work doesn’t produce a number you can put in a press release next to a competitor’s number.
The Leaderboard Is the Marketing
That’s really the crux of it. Benchmarks serve a communications function more than a scientific one at this point. They give journalists a hook, give enterprise buyers a shorthand, and give the labs a way to declare victory on a rolling basis without having to define what victory actually means for users.
The irony is that the people least impressed by benchmark scores are usually the ones using these models most heavily - the developers, researchers, and product teams who’ve already learned to calibrate around the gap between what a model scores and what it actually does when someone asks it something real. What that calibration should tell the industry is a question that nobody seems to be in a hurry to answer.