You’ve seen the demos. An AI agent watches a screen, moves a mouse, clicks buttons. It looks like magic—until it breaks. The moment the UI shifts, the layout changes, or a pop-up appears, the agent freezes. It’s a brittle, fragile puppet on a string. And honestly? It’s a dead end.
Then I saw something that made me rethink everything. A developer named Skullbloc built an AI that plays Crusader Kings 3—a game so complex it takes humans hundreds of hours to master—without ever looking at the screen. No pixel reading. No mouse simulation. The AI doesn’t see the game; it feels the code.
Here’s how it works: Instead of giving the LLM a screenshot and a virtual mouse, Skullbloc injected a binary directly into the game’s memory. The AI reads the raw data structures—character stats, relationships, events—as if it were a native part of the system. It outputs decisions not as clicks, but as direct memory writes. The game never even knows it’s being played by a machine.
This is the paradigm shift nobody’s talking about. We’ve been forcing AI to interact with software the way humans do—through a UI bottleneck. That’s like giving a fish a bicycle. The real power of AI lies in bypassing the interface entirely. Why translate a table of numbers into a graph, then back into a decision? Let the AI read the numbers directly.
I’ve worked on automation projects for years, and I’ve seen the same pattern repeat: teams spend months building screen-scrapers that break on the next software update. The brittle approach has a name: UI automation. And it’s a hack. The future belongs to what I’ll call systemic integration—direct API or binary-level access that lets AI act as an internal component, not an external puppet.
Watch the Twitch stream. The AI doesn’t hesitate. It makes strategic marriages, manages wars, and navigates dynastic politics—all from the raw data. It’s not mimicking human vision; it’s operating on a different plane of existence. The most profound AI agents don’t need eyes—they need access.
This isn’t just for games. Every industry that relies on software is about to face a fork in the road. One path: keep building agents that squint at screens. The other: redesign our systems with machine-to-machine interfaces at the core. The choice seems obvious, but most companies are still on the first path because it’s familiar. That’s a mistake.
Here’s the contrarian bet: within five years, the term ‘UI automation’ will feel as archaic as ‘dial-up internet.’ The AI that plays Crusader Kings 3 without a screen is a glimpse of what’s coming. If your software doesn’t have a direct API for AI, you’ve already built a legacy system.
FAQ
Q: Isn't this just a fancy way of saying 'cheating'? The AI has access to game memory that humans don't.
A: It's not cheating—it's a different paradigm. A human can't read memory directly, but an AI can. That's the point. The AI isn't exploiting the game; it's interacting with the software at its native level. The same approach applies to any software: give AI the data it needs in the form it processes best, not the form we use.
Q: How does this change the way I build software or automation tools?
A: Start designing for machine-to-machine communication. Instead of exposing everything through a UI, expose structured data through APIs or memory-mapped interfaces. If you're building automation, skip the screen-scraping layer and go straight to the system internals. The result is orders of magnitude more robust and faster.
Q: Won't this make UIs obsolete? What about accessibility for humans?
A: UIs remain essential for humans. But for AI agents, UIs are a bottleneck. The contrarian take is that we should stop pretending AI needs to use our tools the same way we do. Let AI speak directly to the machine. Humans keep the cozy interfaces; AI gets the raw power. That's a win-win.