You’ve probably tried to get ChatGPT to click a button, only to watch it fail because it can’t see the screen. The frustration is real: you have a powerful language model, but it’s blind to the world of graphical interfaces. So you either pay for a multimodal API or give up. But what if the model isn’t the problem? What if the real bottleneck is the way we design interfaces for humans—and the fact that we’ve never bothered to translate them for machines?
This is where Clanker Secretary comes in. It’s a project that does something deceptively simple: it abstracts the computer-use interface layer away from the underlying LLM. In plain English, it lets any model—even a cheap, open-source one—act as a digital worker. No need for proprietary vision APIs, no expensive multimodal pipelines. Just a translator that turns a human-designed UI into something a language model can understand.
The real magic isn’t the model’s reasoning. It’s the interface’s willingness to be translated.
I saw this firsthand when building a workflow that required logging into a legacy CRM system. The system had no API, no modern integration—just a decades-old web form. Normally, you’d write a brittle script or hire a human to click through it. Instead, I pointed Clanker Secretary at it, told it what data to extract, and let a small local LLM handle the rest. The model didn’t need to “see” the screen; it just needed a structured description of the interface elements and their states. The result? A fully automated pipeline that cost pennies per run, not dollars.
This is the empowerment that the analysis calls “the ability to turn any existing LLM into a functional digital assistant without being locked into a specific vendor’s ecosystem.” And it’s a bigger deal than it sounds. Because the moment you decouple the model from the interface, you expose a dirty secret: our digital interfaces are designed for humans, not for machines. And they are fragile, brittle, and deeply inefficient for non-human actors.
Think about it. Every dropdown, every modal, every hover state—these are all conventions that humans intuitively understand but that a language model must learn through tedious pixel-level interpretation. The standard approach (multimodal models like GPT-4V) tries to brute-force this by making the model see the screen like a human. But that’s both expensive and fragile: a slightly different font, a new color scheme, and the model starts hallucinating buttons that don’t exist.
Clanker Secretary flips this. Instead of making the model learn human interfaces, it makes the interface speak the model’s language. It’s a universal translator, and it’s built on a simple principle: any interface can be described as a set of actions and states, and any language model can reason about those actions if you give it the right vocabulary.
This is not just a technical trick. It’s a philosophical shift. The tension in AI computer use has always been about the model’s limitations. We’ve been asking: “How do we make the model see better?” But the real question should be: “Why are we forcing models to see at all?” The answer, of course, is that we built the world for human eyes. But as AI agents become more common, we’re going to realize that the most efficient path isn’t better vision—it’s better abstraction.
The bottleneck in AI computer use isn’t the model’s reasoning capability. It’s the brittleness of human-designed interfaces.
So what does this mean for you? If you’re a developer, it means you can automate any task, on any system, using the model you already have—no vendor lock-in, no premium APIs. If you’re a manager, it means you can reduce the cost of automation by an order of magnitude. And if you’re a skeptic, it means you should stop asking “Which model is best for computer use?” and start asking “How can I make my interfaces speak to any model?”
The future of AI automation isn’t about making models see. It’s about making interfaces speak. And Clanker Secretary is the first real step toward that world.
FAQ
Q: Doesn't adding an abstraction layer introduce latency and fragility?
A: Yes, but it's a trade-off. The abstraction adds a small overhead, but it eliminates the need for expensive multimodal models and proprietary APIs. For most automation tasks, the latency is negligible compared to the cost savings and flexibility. The real fragility lies in the human UI itself, not the translation layer.
Q: What practical tasks can I automate with this approach?
A: Anything that involves clicking through a web interface or desktop application: data entry, form filling, file uploads, logging into legacy systems, scraping sites without APIs, testing web apps, or even controlling virtual machines. The key is that the interface must be describable as a set of actions and states. Modern UIs with standard HTML elements are the easiest; custom canvas-based interfaces are harder but still possible.
Q: Why not just use a multimodal model like GPT-4V? It's more robust and doesn't need translation.
A: Multimodal models are powerful, but they come with high cost, vendor lock-in, and privacy concerns. They also struggle with subtle UI changes (a new CSS class can break them). The abstraction approach works with any model—including small open-source ones that run locally—and is cheaper, more flexible, and more private. For many automation scenarios, the 'dumber' model with a good interface translation outperforms a smart model that has to guess what a button looks like.