You’ve been there. You update your AI model, expecting magic. Instead, it starts miscounting the number of objects in an image—something it nailed last week. You’re not crazy. The numbers are real.
A new benchmark called the Little Dorrit test tracks exactly this. It’s a vision task: convert handwritten editor marks on a printed page into structured JSON. Simple, right? Not for your AI.
Fable 5 scores 0.65—the top. But Opus 4.8? It regressed 55% on miscounting errors. The model that can write poetry, solve math problems, and generate code can’t count how many circles are crossed out.
“Every new AI model is a gamble: you gain a wizard, but you might lose a simpleton.”
This isn’t a bug. It’s a feature of how modern AI development works. We treat progress as a linear upward slope, but it’s actually a zero-sum reallocation of resources. Every new capability—better reasoning, larger context windows, multimodal understanding—comes from cannibalizing the neural network’s capacity for the basics. The same weights that learn to write Shakespeare forget how to count to five.
You’ve probably noticed this in your own workflow. The AI that used to flawlessly parse invoices now hallucinates totals. The chatbot that never misread a date suddenly swaps months. It’s not your imagination. It’s the silent regression that nobody in the marketing blog posts talks about.
Why does this happen? Because model updates are optimized for aggregate benchmarks. If a model scores higher on 100 complex reasoning questions but drops 20% on a simple counting task, the update ships. The nuance of your specific use case—the one that depends on that counting—is invisible to the metrics.
“Aggregate improvement is the biggest lie in AI. Your individual use case doesn’t care about the average.”
So what do you do? You stop trusting version numbers. You build your own narrow benchmarks. You test the exact tasks that matter to your production pipeline—every single update. The Little Dorrit test exists because one developer cared enough to catch the regression. You need your own Little Dorrit.
Don’t let the shiny demos fool you. The AI that can write a sonnet may no longer be able to count the words in it. And that’s a problem you can’t afford to ignore.
FAQ
Q: Aren't these regressions just edge cases that don't matter in practice?
A: No. If your workflow depends on counting objects, reading handwritten marks, or parsing structured data, a 55% regression is catastrophic. Edge cases become core failures when you rely on them daily.
Q: How can I protect my production systems from silent regressions?
A: Build a narrow, automated benchmark that tests exactly the tasks your application uses. Run it on every new model version before deploying. Treat AI updates like any software dependency—test before you upgrade.
Q: Isn't it better to have a model that's smarter overall, even if it loses some narrow skills?
A: Only if those narrow skills don't matter to you. For most businesses, the 'simple' tasks are the ones that generate revenue. A model that can reason in complex ways but can't count correctly is a liability, not an upgrade.