The Dirty Secret of AI Agent Benchmarks: It’s Not the Model, It’s the Harness
A new benchmark paper reveals a dirty secret: swapping evaluation harnesses can boost AI agent scores as much as upgrading an entire model. Most ‘model improvements’ are actually measurement infrastructure improvements. The field is partly measuring its own tools—and that changes how we should read every leaderboard.