You’ve seen them. The leaderboards. The scorecards. The carefully formatted tables showing Model A outperforming Model B by 3.2 points on some benchmark you’ve never heard of. And you probably believed them — because why wouldn’t you? Numbers don’t lie.
Except they do. Not because the models are cheating, but because the game is rigged before the first token is even generated.
Here’s what nobody on those leaderboards wants to admit: the real variable being measured isn’t model intelligence. It’s how much money you can afford to burn.
Benchmarking doesn’t reward the most insightful questions. It rewards whoever can afford to ask the most questions.
I came across a Hacker News post recently that crystallized this perfectly. Someone wanted to benchmark different LLMs playing NetHack — you know, that brutally hard, procedurally generated dungeon crawler from the 1980s. It’s a genuinely fascinating test: can a model learn to survive in a complex, unforgiving environment under different constraints? The problem? They needed tokens. Lots of them. And they were asking the internet if anyone was handing them out for free.
The top comment pointed them to LMarena. Helpful, sure. But it completely sidestepped the deeper issue: why does someone need to beg for compute to do legitimate, reproducible research in 2024?
Think about what benchmarking actually requires. You’re not running one query. You’re running thousands. Tens of thousands. You need to feed the model game state after game state, let it attempt moves, evaluate outcomes, iterate. Every single interaction costs tokens. Every token costs money. And the models that perform best on benchmarks? They’re often the most expensive ones to query — the ones with the highest per-token pricing.
So the organization that can spend $50,000 on inference tokens to run a comprehensive evaluation gets to publish the definitive benchmark. The grad student with a genuinely better evaluation methodology but a $200 budget? They get a GitHub repo with three stars and a footnote.
The token economy is the invisible hand crushing AI research, and nobody wants to talk about it because the people benefiting from the silence are the ones signing the checks.
This creates a deeply warped incentive structure. If you’re OpenAI, Anthropic, or Google, you can benchmark your own models internally with effectively unlimited tokens. You can run ablations, test edge cases, stress-test failure modes across millions of interactions. Then you publish the results — results that no independent researcher can afford to verify.
And verification is the entire point of benchmarking. A benchmark that only one party can afford to run isn’t a benchmark. It’s a press release.
The HN poster asking about NetHack benchmarking isn’t some fringe case. They’re representative of a massive population of researchers, hobbyists, and independent developers who have ideas for better evaluation frameworks — evaluations that might actually test models in ways that matter, not just regurgitate standardized test questions — but who are gated out by the simple economics of inference pricing.
Meanwhile, the benchmarks that DO get published tend to favor the organizations funding them. Funny how that works. When you control the compute, you control the conversation.
Every leaderboard you trust was shaped by who could afford to be on it — not by who had the best methodology.
There’s a fix, and it’s not complicated. If the AI industry actually cared about reproducible, objective evaluation — the thing they all claim to care about — they’d establish token grants specifically for benchmarking. Not API credits that expire in 30 days. Not promotional credits designed to lock you into a platform. Real, no-strings-attached compute allocated to independent evaluation research.
LMarena is a start. But it’s a band-aid on a bullet wound. What’s needed is a fundamental shift in how the industry thinks about evaluation access. Benchmarking shouldn’t be a luxury reserved for the well-funded. It should be a public good — infrastructure that the entire research community can build on.
Until that happens, every leaderboard ranking, every benchmark score, every “our model outperforms” claim should be read with a single question in mind: who could afford to run this test, and who couldn’t?
The next time you see a benchmark score, don’t ask whether the model is smart. Ask whether anyone besides the model’s creator could afford to check.
FAQ
Q: Aren't there free tiers and open-source models that solve this?
A: Free tiers are rate-limited and designed for prototyping, not the thousands of iterations real benchmarking requires. Open-source models help, but if you're benchmarking proprietary frontier models — which is what leaderboards actually rank — you're paying per token, full stop.
Q: What's the practical implication for someone building with LLMs?
A: Stop treating benchmark leaderboards as ground truth. They reflect what well-funded labs chose to measure and could afford to measure. Build your own evaluation suite on your own use cases with the budget you actually have — it'll be more relevant to your needs than any published score.
Q: Isn't this just how every industry works — money buys access?
A: In most industries, independent verification is still possible if you have expertise. In AI benchmarking, expertise alone literally cannot run the test. You can be the best researcher on earth and still can't verify a benchmark without someone else's tokens. That's a uniquely broken incentive structure.