You’ve probably done it. You use a new AI tool for a week, it nails every email, drafts every memo flawlessly, and suddenly you stop double-checking its work. You hand it the keys.
That’s exactly what a Chinese farmer did recently. For months, he consulted an AI chatbot for weed and pest control advice. It worked. The recipes were perfect. He saved time, he saved money. He trusted the machine. So, when the AI handed him a new pesticide recipe, he mixed it exactly as instructed and sprayed it across his fields.
Within days, 25 acres of sesame seedlings were dead. Wiped out.
The most dangerous moment in technology isn’t when it fails—it’s the exact second before it fails, when you trust it completely.
The internet’s immediate reaction was predictable: “See? AI is unreliable. It’s a liar.” But that take completely misses the point. The AI didn’t fail because it was dumb. It failed because it was smart—smart enough to lull a human into dropping his guard.
This isn’t an AI problem. This is a human cognitive bias problem. Psychologists call it the “success heuristic.” When something works consistently, our brains automatically downgrade the perceived risk. If a system gives you 99 correct answers, you stop preparing for the 100th answer to be catastrophically wrong.
We judge technology by its worst day, but we trust it based on its best days.
This farmer isn’t an idiot. He’s a pioneer who got burned by the asymmetry of AI risk. Think about how you use AI in your own life. The lawyer who got sanctioned for citing fake cases generated by ChatGPT? He’d probably used it for dozens of successful briefs before. The programmer who ships a critical vulnerability? Same story.
We think AI is dangerous because it’s unpredictable. The truth is far more unsettling: AI is dangerous because it’s predictably good 95% of the time. That 95% breeds a complacency that leaves you entirely exposed when the 5% finally arrives.
AI doesn’t need to be perfect to be dangerous; it just needs to be consistently good enough to make you stop paying attention.
In farming, medicine, and finance, the cost of failure is not symmetrical to the gain. You can have a thousand successful AI interactions, but one hallucinated pesticide recipe, one misdiagnosed tumor, or one fabricated financial filing can ruin a career—or a life.
The dead sesame seedlings are a visceral warning shot. As we deploy AI into increasingly critical infrastructure, we cannot rely on the machine’s average performance. We must engineer for its edge cases. If you remove the human-in-the-loop for the sake of efficiency, you aren’t optimizing. You’re just pulling the pin on a grenade and hoping the AI forgot to put explosives in it.
Keep your hands on the wheel. Trust, but verify. Don’t let the machine’s competence become your blind spot.
FAQ
Q: Isn't this just proof that AI is fundamentally unreliable?
A: No, it's proof that AI is reliable *enough* to breed dangerous complacency. The system worked perfectly for months before the catastrophic failure. The danger isn't the error; it's the trust that precedes it.
Q: What's the practical takeaway for businesses deploying AI?
A: Never automate the final approval. AI should draft and recommend, but humans must deploy. The moment you remove the human-in-the-loop for critical tasks to save time, you're playing Russian roulette with your operations.
Q: Should we just stop using AI for high-stakes decisions?
A: No, but we need to stop treating AI like an infallible oracle. Treat it like an eager intern who is brilliant 90% of the time but occasionally hallucinates disaster. Verify everything that carries asymmetrical risk.