You’ve probably noticed a new kind of wall popping up on websites you used to browse freely. A little puzzle. A proof-of-work challenge. A cookie that says, “Prove you’re human before you can read this.” It’s called Anubis, and it’s being sold as a way to stop AI companies from scraping the open web. But here’s the dirty secret that nobody in the anti-scraping camp wants to admit: Anubis doesn’t stop AI companies. It stops you.
Let me paint a picture. I’m a developer building a small open-source tool that pulls the latest patches from a Linux kernel mailing list. Nothing nefarious. Just a hobby project that saves me time. One day, my Python script stops working. The server returns a 403. I check the logs — Anubis is blocking me because my tool doesn’t solve its proof-of-work challenge. Meanwhile, the big AI labs with their GPU clusters and datacenter budgets are still scarfing up the same data without breaking a sweat. The exact adversary Anubis targets defeats it trivially.
How does this work? Anubis requires a client to compute a moderately hard hash before it serves the content. The idea is to raise the marginal cost of scraping so high that it becomes economically unviable for mass crawlers. Sounds reasonable, right? Except the economics don’t play out the way the designers imagined. For a well-funded AI company, the cost of a few extra GPU cycles is pocket change. For an independent researcher or a hobbyist, it’s a dealbreaker.
This is the paradox of defending the open web by building walls. Every layer of friction you add to stop the bad guys also slows down the good guys. But the asymmetry is brutal: the bad guys have unlimited resources; the good guys are just trying to build something. The result is a system that hands a monopoly on web data to the very entities it claims to block.
Take a look at the comments on the original analysis. One user nails it: “Exactly, it’s the same as Cloudflare captchas where only certain blessed devices and browsers can seem to actually pass it. Ironically, adding Anubis accelerates the death of the open web.” Another points out the flaw: “Anubis stops mass crawling of web pages, by requiring a proof of work, which makes accessing these websites more expensive. … Anubis was not built to stop individual users with LLMs.” But that’s the whole problem — the line between “mass crawling” and “legitimate individual use” is blurry, and the collateral damage is enormous.
So who does Anubis actually stop? It stops the open-source developer who wants to train a model on public data. It stops the academic who wants to run a large-scale analysis of web trends. It stops the privacy-conscious user who uses a custom browser or a VPN. It stops the tiny AI startup that can’t afford a datacenter. It stops the very people who keep the web vibrant and decentralized. By raising the cost of access, we don’t stop the AI giants — we just ensure that only they can afford to play.
This isn’t a bug; it’s a feature of the business model. The companies that deploy Anubis often have their own AI ambitions or partnerships. They’re not protecting the open web; they’re building a moat around their own data. And the rest of us are left outside, locked out by a system that was supposed to be our shield.
I’m not saying we should let AI companies scrape everything with impunity. There are better ways — rate limiting based on user agent patterns, selective blocking of known datacenter IPs, transparent robots.txt enforcement, and legal frameworks that don’t require technical circumvention. But Anubis is not one of them. It’s a lazy, one-size-fits-all solution that punishes the little guy while leaving the giants untouched.
Here’s the uncomfortable truth: If you build a wall that only the poor cannot climb, you haven’t built a wall — you’ve built a ladder for the rich. The web was born open. Every time we add a proof-of-work gate, we’re not defending it. We’re burying it alive. The next time you see an Anubis challenge, ask yourself: Who is this really stopping? The answer might make you want to close the browser tab for good.
FAQ
Q: Doesn't raising the cost of scraping make it harder for AI companies to scrape?
A: Only if the cost is high enough to be a meaningful fraction of their budget. For a company like OpenAI or Google, the extra compute cost of a proof-of-work challenge is negligible. For a hobbyist, it's a showstopper. The asymmetry is the key failure.
Q: What's the practical alternative to Anubis if we want to protect content from AI scrapers?
A: Better approaches include transparent rate limiting, IP-based blocking of known datacenter ranges, selective serving of content to verified human users via browser fingerprinting, and legal frameworks like copyright enforcement that don't rely on technical barriers. None are perfect, but they don't create a two-tier web.
Q: Is the open web really dying, or is this just techno-pessimism?
A: It's dying in slow motion. Every new layer of friction — CAPTCHAs, proof-of-work, login walls, browser fingerprinting — chips away at the principle of universal access. The web was designed to be a level playing field. Tools like Anubis are turning it into a gated community where only the wealthy can afford unrestricted entry.