You’ve watched it happen. You build an AI agent. It demos beautifully. Your boss is thrilled. Then it goes to production and sets a customer’s order total to negative $4 million because it confidently hallucinated a discount code that never existed.
And everyone stands around pretending this is just a “version one” problem.
It’s not. It’s an architecture problem. And it’s going to keep happening until we stop treating large language models like they’re general-purpose computers.
Every time you ask an LLM to do something a deterministic system can do reliably, you’re not innovating. You’re gambling with someone else’s money.
Here’s the paradox nobody in the AI agent space wants to talk about: we want our agents to be autonomous and intelligent, but their very nature — probabilistic, next-token-predicting machines — makes them fundamentally unreliable for any task that demands guaranteed correctness. You can’t fine-tune your way out of this. You can’t guardrail your way out of this. The model isn’t broken. You’re just using it wrong.
Think about what happens when you ask an LLM to calculate a price. It doesn’t compute. It predicts. It looks at the numbers and the context and guesses what the answer should look like. Sometimes — often, even — it gets it right. But “often right” is a catastrophic failure mode when the task is “charge this customer’s credit card.”
The real leverage isn’t in making the LLM smarter. It’s in re-architecting the entire system so the LLM never touches deterministic operations in the first place.
Treat the LLM as a fallible advisor, not a central controller. Let it decide what should happen. Never let it decide what did happen.
This means something specific and uncomfortable for a lot of teams: your AI agent should be surrounded by boring, traditional software. Calculations go to a calculator. Database queries go through a validated ORM. Business rules execute in code that was tested six ways to Sunday. The LLM’s job is to understand intent, navigate ambiguity, and choose which deterministic tool to invoke — then step back and let the tool do its job.
One commenter on the original piece that inspired this thinking put it perfectly: a clever agent system would create deterministic building blocks that can be assembled at a fraction of the cost of having the model reason through everything from scratch. The model explores. The scaffolding executes.
This isn’t a minor optimization. It’s the difference between a product that scales and a product that generates a new incident report every Tuesday.
I saw this firsthand with a team that built a customer support agent. Version one: the LLM handled everything, including refund calculations. It worked 94% of the time. That sounds great until you realize 6% of refunds were wrong — and 6% of a million transactions is 60,000 angry customers. Version two: they ripped out every calculation, every API call, every state transition, and replaced them with deterministic functions the LLM could call. The agent’s accuracy didn’t improve. Its reliability became a non-question, because the model was no longer responsible for correctness — only for routing.
The model doesn’t need to be right about the answer. It needs to be right about which system has the answer.
If you’re a developer, product manager, or tech strategist building LLM-powered products right now, this is the insight that separates the demos from the deployments. The companies winning with AI agents aren’t the ones with the smartest models. They’re the ones with the best boundaries around them.
So stop asking your model to do arithmetic. Stop asking it to format JSON. Stop asking it to enforce business rules. Give those jobs to the systems that were built for them — systems that don’t hallucinate, don’t drift, and don’t need a 47-billion-parameter safety net to get basic math right.
The future of AI agents isn’t more intelligent models doing more things. It’s intelligent models doing less — and doing that less with terrifying precision.
The best AI agent architecture is the one where the AI does the least amount of damage.
FAQ
Q: But what about agentic frameworks that let LLMs write and execute code? Aren't they solving this?
A: They're masking it. Code-generating agents still rely on the model to decide what code to write and when to run it. That's still probabilistic control over deterministic execution. The fix isn't letting the model write code — it's giving it pre-written, pre-tested functions to call. Let it choose. Don't let it build.
Q: What does this mean for my team's roadmap?
A: Audit every place your LLM touches a deterministic operation — calculations, state transitions, data formatting, business rule enforcement. Rip those out and replace them with traditional code exposed as tools. Your agent gets simpler, your system gets reliable, and your incident count drops overnight.
Q: Isn't this just saying LLMs are useless?
A: No — it's saying they're narrowly useful, and that's a feature, not a bug. LLMs are extraordinary at understanding messy human intent and routing it to the right system. That's a massive capability. But the moment you ask them to also be the system, you've created a reliability nightmare disguised as innovation.