That AI coding assistant that claims 90% on SWE-Bench? It’s lying to you. Not with malice, but with math. The kind of math that makes you feel crazy when you actually use the tool and it forgets your context fifteen minutes in.
You’ve experienced this. You’re working on Feature X. Everything clicks. Then a spot bug appears. You switch to fix it. Five minutes later, you return to Feature X. The AI has forgotten the plan. Cache expired. Context gone. You start over. The brochure didn’t mention this part.
If you’re evaluating AI on isolated tasks, you’re rewarding tools that are great at one-shot puzzles. But that’s not what software development is. Software development is a messy, multi-quarter game where you need to dribble, pass, defend, and think on your feet. It’s not a free-throw contest.
A developer named Matt got tired of the lie. He started building a better benchmark. His idea is brutally simple: don’t test isolated tasks. Test sessions. Stitch together ten SWE-Bench verified tasks from the same repo into one long session. Start with the hard one. Measure cost, quality, and turn count.
This changes everything. Suddenly, the tool that scores 90% on ‘one-and-done’ tasks might collapse when asked to just… keep going. Cache TTLs expire. Context grows. You change your mind. The AI has to come along for the ride.
The real metric isn’t “What’s its MMLU score?” It’s “How good is it at context management?”
This isn’t academic. If you are a CTO buying AI tools for your team, you are making a bad bet. You are buying a benchmark score that doesn’t predict real-world performance. You are paying for a tool that looks great in the lab but gets lost in your codebase.
The community is already fixing this. Tools like stet.sh mine tasks from your merged PRs. They replay them in Docker containers. They test AI the way we actually work: with real code, real mess, real context.
It’s time to stop testing AI with exams and start testing it with actual work. The tools that survive a session-based benchmark aren’t just smart. They’re reliable. And in the messy reality of software development, reliability beats brilliance every single time.
FAQ
Q: Isn't a session-based benchmark just adding unnecessary complexity? Isn't a simple one-shot pass/fail easier and more objective?
A: Easier, yes. Objective, no. A simple benchmark gives you a false sense of confidence. It measures the wrong thing. A session benchmark is harder to design, but it measures what actually predicts real-world utility: context management, cost efficiency, and adaptability. It's the difference between a lie and a useful signal.
Q: If I'm a CTO, how do I actually apply this 'session benchmark' today?
A: Stop relying on SWE-Bench scores alone. Demand session-based results from vendors. Run internal trials that simulate your actual workflow—not isolated tickets, but a sequence of related tickets from your backlog. Measure cost vs. quality to completion. The tools that pass this test will save you money. The ones that fail will waste it.
Q: Is it possible that some AI models are genuinely better at session workloads, or is it just a data artifact?
A: It's a real architectural difference. Tools optimized for context retention, long-range attention, and cost-efficient caching will inherently perform better on sessions. A tool that 'cheats' by being fast but shallow on isolated tasks will fail the session test. This isn't about gaming a new metric. It's about revealing a fundamental capability that was previously hidden.