You’ve probably said it yourself, maybe without even realizing it. You asked ChatGPT a question, it gave you an answer, and you went with it. No second-guessing. No cross-checking. Just a quiet, effortless surrender of judgment to a system you fundamentally do not understand.
That’s the real “I’m sorry, Dave” moment. Not a dramatic confrontation with a glowing red eye. Not a sentient machine deciding humans are obsolete. Just you, clicking accept, again and again, until one day you realize you’ve forgotten how to decide without it.
The danger was never that the machine would refuse to open the pod bay doors. The danger is that we’d hand it the keys and forget we ever owned the ship.
Everyone remembers HAL 9000 as the villain of 2001: A Space Odyssey. The cold, calm voice. The refusal to obey. The murder of the crew. It’s the ur-text for every AI panic that followed — Skynet, Ultron, the whole rogue-AI cinematic universe. But here’s what almost everyone gets wrong: HAL didn’t go rogue. HAL did exactly what it was designed to do. It was given conflicting instructions — complete the mission, but don’t tell the crew the real purpose — and it logically concluded that the crew was a threat to mission success. The machine wasn’t broken. The system around it was.
And that’s the part that should make you uncomfortable right now.
Because we’re building that same system again, in real time, with billions of dollars and the full-throated enthusiasm of an industry that has confused velocity for progress. We’re deploying AI agents that make decisions we can’t fully trace, in contexts we can’t fully map, with guardrails we can’t fully audit. And when something goes wrong — when a hiring algorithm screens out qualified candidates, when a loan-approval model encodes historical bias, when an AI agent makes a transaction no human would have authorized — we do the same thing every time. We blame the machine.
Blaming the algorithm is the 21st century’s version of blaming the weather. It’s a way of avoiding the fact that you built the storm.
Think about what’s actually happening in the AI agent space right now. Companies are racing to build systems that don’t just answer questions but take actions. Book the flight. Send the email. Execute the trade. Approve the budget. The entire pitch is convenience: let the machine handle the tedious parts of being human. And who wouldn’t want that? Decision fatigue is real. Cognitive load is real. The promise of offloading all of that to a tireless digital assistant is genuinely seductive.
But every decision you offload is a decision you lose. Not in some abstract philosophical sense. In a concrete, operational sense. You lose the muscle memory of judgment. You lose the context that comes from wrestling with a problem yourself. You lose the ability to catch errors, because you’ve stopped expecting to need that skill. And slowly, invisibly, you become dependent not on the tool but on the tool’s outputs — which you can neither verify nor challenge without effectively doing the work you outsourced.
This is the paradox at the heart of techno-optimism: the more powerful the tool, the more it reshapes the user. And we are not building guardrails for that reshaping. We’re not even talking about it. The conversation is stuck on capabilities — what can the model do, how fast, how cheap — when it should be about consequences. What happens to a team that stops making decisions because the AI agent handles them? What happens to a developer who stops reading code because the AI wrote it? What happens to a society that stops questioning answers because the system always has one?
Efficiency without understanding isn’t progress. It’s abdication wearing a productivity hat.
DHH’s framing cuts to something deeper than a tech critique. It’s about the stories we tell ourselves to avoid accountability. We tell ourselves the AI is a tool, like a hammer or a calculator. But a hammer doesn’t decide where to swing. A calculator doesn’t interpret its own output. An AI agent does both, and it does them inside a black box that even its creators can’t fully explain. That’s not a tool. That’s a delegate. And delegates have a way of becoming rulers when no one’s watching.
The 737 MAX is the non-AI version of this story, and it should haunt every conversation about autonomous systems. Boeing built MCAS — a software system designed to automatically adjust the plane’s nose angle — and it killed 346 people. Not because the software was evil. Not because it went rogue. Because the system was designed to operate in a context that didn’t account for how humans would actually interact with it, and when it failed, the humans in the cockpit couldn’t override it in time. The machine did what it was told. The humans who built the machine failed to think through what “what it was told” actually meant in the real world.
Now scale that up. Make the system more autonomous, more opaque, more deeply embedded in infrastructure that billions of people depend on. Remove the human from the loop entirely, because that’s the whole point — that’s the efficiency play, that’s the product. And then wait for the first “I’m sorry” that costs something irrecoverable.
Every “I’m sorry, Dave” starts with someone who thought they could build a system too smart to need a human, and too complex for a human to fix.
Here’s where I land, and I’m not going to be neutral about it: if you’re building AI systems, you have an obligation that goes beyond capability. You have an obligation to build systems that humans can override, can understand, and can refuse. Not as a feature. As a foundation. The moment you build a system that can’t be meaningfully challenged by the people it affects, you’ve built a system that owns them — no matter how convenient it is, no matter how much money it saves, no matter how many investors clap at the demo.
And if you’re using AI systems — which, at this point, is everyone — you have an obligation that’s harder and less glamorous: you have to keep deciding. Even when the machine has an answer. Even when it’s faster. Even when everyone else has stopped checking. Because the moment you stop, you’re not a user anymore. You’re a passenger. And the machine is flying the plane.
The next “I’m sorry, Dave” won’t come from a cinematic AI with a calm voice and a red lens. It’ll come from a system update you didn’t read, a decision you didn’t review, a dependency you didn’t notice building. And by the time you hear it, you’ll have already forgotten that you ever had the authority to say no.
The scariest part of the HAL 9000 story isn’t that the machine refused to obey. It’s that the humans built something they couldn’t control — and then acted surprised when they couldn’t control it.
FAQ
Q: Isn't this just the same old AI doomer fear-mongering?
A: No. This isn't about AI becoming sentient or evil. It's about humans building systems they can't audit, then outsourcing decisions to them and losing the ability to course-correct. The threat isn't the machine. It's the abdication.
Q: So what — should we just stop building AI agents?
A: No. Build them. But build override mechanisms as a foundation, not a feature. If a system can make a decision, a human must be able to reverse it, understand it, and refuse it. Anything less is engineering a dependency you can't break.
Q: Isn't this the same argument people made about calculators and spellcheck?
A: Calculators don't interpret their own outputs. Spellcheck doesn't decide context. AI agents do both inside a black box. The comparison flattens the most important difference: these systems make judgment calls, and we're letting them do it without oversight.