You’ve seen the headlines. A new AI model drops, it tops the SWE-Bench leaderboard, and the tech press declares we’re one step closer to AGI. You eagerly plug it into your workflow, ask it to build a simple component, and… it hallucinates a library that hasn’t existed since 2018.
We are trapped in an arms race of ‘benchmaxxing’—where AI labs aren’t necessarily building smarter systems, they’re building systems that are terrifyingly good at taking standardized tests.
A benchmark is just a test you can memorize. True intelligence is figuring out the test you’ve never seen before.
The dirty secret of the AI industry is that public benchmarks are essential for measuring progress, but they inevitably become targets to optimize for. The moment a benchmark is published, it ceases to be a measure of general capability and becomes a narrow track for overfitting. The gap between apparent improvement and genuine capability widens every single day.
Don’t believe me? Look at the real-world tests happening in the trenches. One developer recently tried to force frontier models to replicate an original 1990s Super Soaker prototype design using standard hardware store parts. Total failure. Another developer is using build123d to generate functional 3D parts from text descriptions alone. The models fail spectacularly. Why? Because you can’t cheat a Super Soaker test. You either understand fluid dynamics and hardware constraints, or you don’t.
Most people focus on which model tops a leaderboard. But if you actually build or rely on AI coding tools, the real value lies in private, domain-specific evaluations that expose the failure modes public benchmarks miss.
The leaderboard doesn’t care if your app crashes in production. It only cares if the model passed a sanitized, pre-approved test.
If you want to know if an AI model is actually useful, stop looking at the public scores. Build your own private benchmark. Throw your messiest, most unstructured, undocumented legacy code at it. Ask it to build something physical, or functional, or weird. That is where the signal lives.
Stop chasing the leaderboard. Start chasing the failure modes.
Achieving AGI won’t be about passing every benchmark ever created. It will be about accounting for the unknown problems, the messy realities, and the Super Soakers of the world. Until a model can handle the unknown, the public leaderboards are just noise.
FAQ
Q: Aren't benchmarks still useful for tracking basic progress?
A: Yes, but only as a floor, not a ceiling. They prove a model isn't completely broken, but they don't prove it's capable of real, unstructured work.
Q: What's the practical implication for developers?
A: Stop trusting the hype. Build your own private, domain-specific tests. Throw your messiest, undocumented code at the model. If it survives your private hell, it's ready for production.
Q: What's the contrarian take?
A: Public AI leaderboards are actively harming progress. They incentivize labs to build test-taking machines rather than useful tools, tricking the market into adopting fundamentally flawed models.