You know that feeling at 3 AM when the pager goes off? Your heart races, you stumble to your laptop, and you pray the dashboard tells you exactly what’s broken. For years, we’ve wanted AI to take that pain away. And now, it finally is. AI is handling incidents, writing the runbooks, and fixing the alerts. It feels like magic. But it’s actually a trap.
Resilience and automation aren’t holding hands; they’re locked in a zero-sum death match.
We thought AI would handle the mundane, freeing us to do the “real” engineering. Instead, it’s quietly stripping away our hands-on intuition. As AI absorbs routine incident response, we lose the very tactile understanding required to handle exceptional failures. The better AI gets at handling incidents, the less humans touch the system. And the less we touch it, the less capable we become when the AI inevitably fails.
Most people frame this as a skills degradation problem. It’s not. It’s an institutional amnesia problem. Tacit operational knowledge stops being created altogether. Future engineers won’t just be rusty—they will have absolutely no mental model to fall back on for systems they inherited. It’s like inheriting a massive codebase where every line was written by a machine, and looking at it feels exactly like your first day on the job. The loss of intuition is a seed of technical debt that grows until it chokes the entire system.
You aren’t just losing a skill; you’re losing the mental map of the very system you built.
The aviation industry learned this the hard way. When autopilot became flawless, pilots lost their manual flying instincts. When the technology failed, they crashed. The tech community loves to reference the 1983 paper “Ironies of Automation,” which predicted exactly this: the more reliable the automation, the more disastrous the human response when it breaks. We are rediscovering this truth in real-time.
It taps into the quiet fear many engineers have: being held responsible for systems they no longer understand, while quietly becoming obsolete in the act of building them. You can give a team the exact solution on a silver platter, but if they lack the operational intuition of how the system breathes, they will still fail to implement it.
We have a choice to make. We can either deliberately force humans to stay in the loop to preserve operational intuition, or we can accept a future where we are merely spectators to our own infrastructure.
The next time AI instantly resolves an incident, don’t breathe a sigh of relief. Ask yourself what you just forgot. Because one day, the AI won’t catch it, and you’ll be the only one left staring at a screen you no longer recognize.
FAQ
Q: If AI can resolve incidents perfectly 99% of the time, why should we care about the 1% of human failures?
A: Because that 1% will be catastrophic. When a complex, novel failure occurs that the AI hasn't seen, the humans in charge will have zero operational intuition to fall back on, turning a manageable crisis into a prolonged outage.
Q: What's the practical implication for engineering teams right now?
A: You must intentionally inject friction. Force engineers to manually handle a percentage of incidents, review AI-generated fixes line-by-line, and participate in chaotic disaster recovery drills to keep their tacit operational knowledge alive.
Q: Isn't this just the same fearmongering we heard when compilers replaced assembly language?
A: No. Compilers abstracted syntax, but engineers still had to understand the logic and state of the system. AI incident response abstracts operational state itself, meaning we lose the mental model of how the living system actually behaves under stress.