You’ve shipped an AI feature. The demo looked great. Your boss was impressed. And now, at 2 AM, you’re lying awake wondering if the prompt you tweaked at 4:55 PM on a Friday actually made things better — or just made them different.
Every developer building on LLMs knows this feeling. You change a word in a system prompt, and suddenly your outputs shift in ways you can’t fully explain. You run a few manual tests. They look fine. You ship it. And then a user finds the edge case you never thought to test.
Here’s the dirty secret of the AI gold rush: most teams are flying completely blind.
The expensive eval suites sold to enterprises are designed for teams who can afford to burn thousands of dollars validating every prompt change. Everyone else — the solo developers, the startup founders, the indie hackers — is left guessing.
Enter Smevals. It’s a collaboration between Simon Willison — the man who’s been documenting the AI revolution longer than most people have been in it — and the developer behind the Superpowers plugin. And it’s not what you think.
Most evaluation tools focus on one thing: testing the model. Does GPT-4 or Claude produce better output for your use case? That’s the question they answer. But Smevals asks something more interesting — something most people miss entirely.
Smevals doesn’t just evaluate your model. It evaluates your evaluator. Your prompt, your harness, your testing framework — the entire stack becomes a recursive feedback system that can be gamed, optimized, or quietly broken.
Think about what that means. The prompt you wrote to test your prompt? That can be gamed. The harness you built to catch failures? That can be optimized into uselessness. The testing infrastructure you trust to tell you the truth about your AI system? It has its own blind spots, its own biases, its own failure modes.
This is the paradox at the heart of AI development. You need small, fast evaluations to validate large, complex models. Full-scale testing is prohibitively expensive — sometimes thousands of dollars per run. But relying on a tiny test suite means accepting that you’re seeing a narrow slice of reality. The risk is enormous. The cost of doing it properly is enormous. There’s been no middle ground.
Until now.
Smevals sits in that gap. It’s lightweight. It’s cheap. It’s the kind of tool you reach for when you want a sanity check, not a certification. It won’t replace your full eval pipeline, but it will catch the stupid mistakes — the ones that cause public embarrassment.
The real danger in AI development isn’t that models are unpredictable. It’s that developers have convinced themselves their five manual tests count as ‘evaluation.’
Here’s what changes when you have a tool like this. You stop treating prompt engineering as guesswork and start treating it as iteration. You make a change. You run a quick eval. You see if it actually improved things. You sleep better.
Willison has been writing about AI tooling with the clarity of someone who actually builds things — not just opines about them from a distance. His collaboration on Smevals reflects a philosophy worth paying attention to: the best tools don’t solve every problem. They solve the problem you actually have, cheaply and quickly, so you can get back to building.
The future of AI development won’t belong to teams with the biggest eval budgets. It’ll belong to the ones who can iterate fastest without breaking things.
If you’re building AI features and you don’t have a lightweight eval suite, you’re not being lean. You’re being reckless. Smevals isn’t a luxury. It’s the difference between catching a regression at 4 PM on a Tuesday and discovering it in a Twitter thread at midnight on a Saturday.
The question isn’t whether you can afford to evaluate your prompts. It’s whether you can afford not to.
FAQ
Q: Isn't a tiny eval suite just a false sense of security?
A: Yes, if you treat it as your entire testing strategy. Smevals is a sanity check, not a certification. It catches regressions and obvious failures — the stuff that embarrasses you publicly. It doesn't replace full-scale evaluation; it makes sure you actually do the cheap, obvious things before reaching for the expensive ones.
Q: What does 'evaluating the evaluators' actually mean in practice?
A: Your test prompts, your harness, and your grading criteria are themselves code that can be gamed or optimized into uselessness. Smevals makes this recursive problem visible — you can see when your eval harness is rewarding the wrong behavior or missing failure modes it should catch.
Q: If big companies already have expensive eval pipelines, why should anyone care about a lightweight tool?
A: Because the future of AI development belongs to fast iterators, not big spenders. The teams that can ship a prompt change, verify it in minutes, and move on will outpace teams waiting on a $5,000 eval run. Speed of iteration beats depth of validation for 90% of real-world use cases.