Last year, I killed an AI Agent project that had made people clap.
Not politely clap. Actually clap. The kind where someone in the back row leans over and whispers, “How did they do that?”
Three weeks of building. One demo. Fifty real users. Fourteen days later — dead. Pulled offline. Eulogized in a Slack thread nobody wanted to read.
Here’s what haunts me: the model wasn’t the problem. The model did everything it was supposed to do. The problem was us — the people who looked at a demo and confused a performance with a product.
If you’re building or funding AI Agents right now, you need to hear this. Because the failure rate for enterprise Agent projects is over 90%, and the way they die is almost identical. They die the same way mine did.
A demo is a greenhouse. Real users are a jungle. And your model can’t tell the difference.
Let me walk you through the three ways we lied to ourselves — because if you’re honest, you’re probably lying to yourself right now in exactly the same way.
Mistake #1: We Designed for a User Who Doesn’t Exist
On demo day, I sat in that conference room and said to the Agent: “Pull last month’s East China sales data, make a weekly report, and send it to Zhang.”
Clean. Precise. Unambiguous. Time specified. Region specified. Action specified. I even knew which database table the data lived in.
The Agent crushed it. SQL written. Charts configured. Email sent. Done.
Then we let real users in. And real users said things like: “Hey, grab that East district stuff from last month and fix it up for me.”
Which East district — East China, or a specific province within it? “Fix it up” means pull the data, or build a report, or send it out? “Last month” — calendar month or their finance team’s definition? The user hadn’t thought it through. He didn’t even know what he wanted.
We had designed for a fictional user who speaks in clean, structured commands. Real users are wild. They change their minds mid-sentence. They use pronouns that point at the wrong thing. They’re emotional, imprecise, and chaotic.
No model — no matter how smart — can bridge that gap, because the gap isn’t a technology problem. It’s a product judgment problem. We took a messy, unpredictable human and simplified them into a button.
Mistake #2: We Fell in Love with Single-Step Success Rates
Here’s a number that will mess with your head.
Say your Agent has a 95% success rate on any single action. Sounds incredible, right? You’d put that in a slide deck. You’d tell your boss the system is rock-solid.
But a real task — the kind users actually need done — isn’t one action. It’s five. Six. Sometimes more. And when you chain them together, the math turns brutal.
0.95 to the fifth power is 77%. That means one in four tasks fails somewhere along the chain. And when one step breaks, everything after it is garbage.
95% accuracy sounds great until you realize that five steps in a row means 77%. That’s not a success rate — that’s a catastrophe wearing a nice outfit.
We learned this the hard way. A user told our Agent to “delete this record.” It deleted — but “this” referred to a different record mentioned three messages ago. The pronoun was wrong. The Agent didn’t hesitate. It executed the wrong action with beautiful, terrifying efficiency.
The user doesn’t remember the four steps you got right. They remember that you sent the weekly report to the wrong person.
Now here’s my rule: anyone who reports a single-step success rate to me has to break down how many steps the full task requires, and multiply those rates together. After the multiplication, nobody bangs the table anymore.
Mistake #3: Nobody Wanted to Answer the Only Question That Mattered
This is the one nobody talks about, and it’s the one that kills projects dead.
Traditional software has a clear accountability chain. Bug in the code? Fix the code. User error? That’s on the user. But an Agent has autonomy. It decides what to do next. So when it decides wrong — when it sends money to the wrong account, deletes the wrong record, places an order with the wrong parameters — who pays?
The model vendor says: “We provide capability. How you use it is your business.”
The product team says: “The user confirmed the action.”
The user says: “You told me the AI could handle this automatically.”
Before our project even went live with real money, we were already fighting about this internally. Legal asked one question: “If it makes an autonomous error, what does the company liable for?”
The room went silent. Nobody had an answer.
The hardest problem in AI isn’t making it smart. It’s answering one question: when it screws up, who pays?
That demo video we all loved — the one where the Agent sailed through every step with zero confirmation prompts — we thought that was elegance. We thought that was the future. Now I realize the most dangerous thing about that video was how smooth it looked.
So Should You Stop Building Agents?
No. I use Agents every single day. I can’t work without them anymore. The 10% that survive are genuinely saving companies money, time, and headcount.
But those 10% survive because someone asked the ugly questions before launch, not after. Here’s what I now demand before any Agent project gets greenlit:
One: Is this scenario fault-tolerant? Internal, read-only, easily reversible — perfect for Agents. External, money-moving, one-strike-and-it’s-a-disaster — stay away. Our project died because we picked the hardest possible scenario on day one.
Two: Stop using demos to deceive yourself and your boss. In the demo phase, actively try to break it. Use the messy, emotional, mid-sentence-pivoting way real users talk. If it survives that, it’s worth building. Demos should show dirty work, not just pretty work.
Three: Keep a human at the trigger. Full autonomy is the dream and the trap. The pattern that actually works: let the Agent do steps one through nine, then hand the final decision — “should this actually send?” — back to a human. Slower. But alive.
Four: Answer the accountability question at立项, not at the incident. Who’s responsible. How is it logged. How do you roll back. What’s the escalation path. These aren’t launch-day checklists — they’re go/no-go criteria.
Five: Track real numbers, not demo applause. Final task completion rate. Human intervention rate. Recovery cost after errors. These numbers are ugly. They’re also true.
The Real Lesson
Here’s what I’ve come to believe after watching my own project die.
Demo culture has rotted our product judgment. A demo is a performance — it exists to show the best possible version of something. But users don’t live in demos. They live in unclear requirements, mispointed pronouns, sudden changes of mind, and the regret of clicking the wrong button.
The products that survive won’t have smarter models. They’ll have people stubborn enough to watch real users fail, curse, and occasionally succeed — and build for that reality instead of the fantasy.
That cost me a dead project to learn. Expensive. But worth it.
Because now, when someone sends me one of those silky-smooth demo videos where the Agent glides through every task without a single confirmation prompt, my first thought isn’t “We can build that.”
My first thought is: “What input made it run that clean? And at which step would my wild, real users make it stumble?”
No demo can answer that for you. Only the jungle can.
FAQ
Q: Isn't the real problem just that the models aren't good enough yet?
A: No. The models do what they're supposed to do. The problem is that we evaluate them in demo conditions — clean inputs, idealized users, single-step tasks — and then act shocked when they fail in messy reality. It's a product judgment failure, not a model capability failure.
Q: So what should I actually do differently when building an Agent?
A: Three things: pick fault-tolerant scenarios first, test with messy real-user inputs during the demo phase instead of after launch, and keep a human at the final decision step. Full autonomy is a trap — the pattern that survives is nine steps of Agent, one step of human confirmation.
Q: You're saying 90% of Agent projects are doomed. Why build at all?
A: Because the 10% that survive are genuinely transformative. The difference isn't luck or better models — it's that someone asked the ugly questions (who's accountable, what's the cumulative error rate, is this scenario fault-tolerant) before launch instead of after the first incident. Demos don't answer those questions. Only real users in real conditions do.