You click a link to read an article, and your browser freezes. Three megabytes of JavaScript boot up. A video ad autoplays. A newsletter pop-up begs for your email. You don’t want the webpage. You just want the words. Now imagine an AI agent trying to do the exact same thing.
The modern web isn’t built for humans or machines. It’s built for advertisers.
Enter the “Accept: text/markdown” proposal. It’s a beautiful, simple idea: servers detect an AI agent and serve a clean, ad-free Markdown file instead of a bloated HTML mess. No scraping. No parsing. No JavaScript bloat. It promises to open the web to AI by removing the friction, creating a clean, machine-readable layer of the internet.
It sounds like a truce in the AI scraping wars. It’s actually a battlefield.
Most discussions treat this as a harmless feature request. It’s not. It’s a trust protocol. And the idea that the biggest AI companies will voluntarily adopt it is dangerously naive.
Any protocol that relies on the honor system is just a vulnerability waiting to be exploited.
Look at the top AI companies—OpenAI, Anthropic, Google. They don’t want a clean, negotiated exchange. They want to scrape everything, hoard it, and control how that content is captured and monetized. If they adopt the Markdown header, they hand over control of the interaction. They lose the ability to train in the shadows.
Their incentives point entirely in the opposite direction. As one developer bluntly put it: “I’ll do that once any of the top 4 AI chatbots says they’ll start making requests with this header. Before that it’s just a neat idea with no adoption.”
But let’s say they do adopt it. What happens then? Without cryptographic or semantic guarantees, it becomes a liar’s paradise. Any malicious agent can lie about its Accept header. Any server can serve poisoned Markdown designed to hijack the agent’s logic. It’s a prompt injection nightmare waiting to happen.
Gatekeepers never voluntarily dismantle the gates. They just build taller ones.
The open internet desperately needs a real solution instead of the current hostile scrape-and-block arms race. We are tired of gatekeepers like Bright Data, Firecrawl, and Cloudflare deciding who gets to access what. But that solution isn’t going to come from a polite HTTP header asking AI giants to play nice.
The real battle isn’t over whether to build the clean channel. It’s over who controls it. And right now, you don’t control a damn thing.
FAQ
Q: What's to stop AI companies from just lying about their Accept header?
A: Absolutely nothing. Without cryptographic or semantic guarantees, any agent can spoof its headers. It's an honor system in an industry built on scraping and subterfuge.
Q: So should publishers even bother implementing this?
A: Not yet. Until at least one of the top AI chatbots publicly commits to using the 'Accept: text/markdown' header, implementing it is just wasted engineering effort.
Q: Isn't this just a way to bypass ads and starve publishers?
A: Yes, and that's exactly why the big AI companies won't adopt it. They want to control how content is monetized; they don't want to accidentally kill the ad-supported web before they can figure out their own revenue model.