You set up a robots.txt file. You wrote the rules. You said “stay out.” And Amazonbot walked right past your sign, opened your cabinets, and started executing code you never offered it.
That’s not a bug. That’s the reality of the web in 2024.
A developer recently set up a honeypot — a fake git repo with fake HTTP endpoints, hidden inside an HTML comment on a public page. Not linked. Not advertised. Buried in source code where no human would stumble on it accidentally. Three weeks later, Amazonbot was hitting those fake endpoints. It hadn’t just scraped the page. It had read the shell script, parsed the endpoints, and fired requests at them.
Your robots.txt is a polite suggestion. Amazon treats it like a junk mail flyer on a doorstep — something to step over on the way in.
Let’s be clear about what happened here. This wasn’t a crawler reading text. This was a crawler executing logic. It followed a trail deliberately hidden from view, interpreted code, and acted on it. That’s not scraping. That’s automated exploration with intent.
The developer’s robots.txt explicitly listed Amazonbot as disallowed. Amazon ignored it. Not because of a configuration error. Not because of a bug. Because robots.txt is a voluntary protocol — a gentleman’s agreement from a kinder era of the internet when search engines still pretended to care about site owners.
That era is dead.
The HN comments tell the real story. One user wrote: “If it’s only Amazon, then you are in a good position.” Another said their small website got killed entirely by AI scrapers. Not Amazon alone — the entire swarm. The crawlers came like locusts, and the site couldn’t survive the traffic. The advice from the community? Ban IP blocks. Block ASNs. Put everything behind Cloudflare. In other words: build a wall, because the front door lock doesn’t work.
There is no law that says a crawler must respect your wishes. The entire framework is built on trust, and the largest companies on earth have decided trust is optional.
Think about the power asymmetry here. You run a personal website. You pay for hosting. You create content. A trillion-dollar company sends its bots to harvest your work, ignore your explicit instructions, and execute code hidden in your pages. You try to contact them. Good luck finding a human. Good luck getting a response that isn’t an auto-reply. Good luck getting anyone to care.
And what’s your legal recourse? Essentially none. Robots.txt has never been legally binding. It was a social contract, and social contracts only work when both parties have something to lose. What does Amazon lose by ignoring your robots.txt? Nothing. What do you lose by being scraped? Bandwidth, server costs, security, and the basic dignity of controlling your own property.
The twist nobody’s talking about: this isn’t just about data theft anymore. When crawlers start executing code — following shell scripts, hitting hidden endpoints — they’ve crossed from reading into acting. That introduces an entirely new threat surface. What if the honeypot had contained malicious payloads? What if a crawler executing arbitrary code opened up vulnerabilities on the crawler’s infrastructure, or on yours? We’re in uncharted territory, and the map hasn’t been drawn yet.
The web was built on the assumption that visitors are guests. Amazon has decided it’s a landlord who can enter any apartment at any time — and you’re just renting.
So what do you do? You stop relying on politeness. You stop trusting protocols that have no enforcement mechanism. You deploy technical countermeasures — IP blocking, ASN bans, Cloudflare bot management, rate limiting, honeypots that trap rather than just observe. You treat every crawler as hostile until proven otherwise, because the evidence is in: the biggest players have already chosen hostility.
The robots.txt file on your website right now is a fiction. It’s a wish written in plaintext to companies that don’t grant wishes. The question isn’t whether they’ll ignore it. They already are. The question is whether you’ll keep pretending it matters — or start building defenses that actually do.
FAQ
Q: Isn't robots.txt legally enforceable?
A: No. It's a voluntary protocol with zero legal weight. No court has ever enforced it as a binding agreement. It works only when crawlers choose to comply — and the biggest ones increasingly don't.
Q: What should I actually do to protect my site?
A: Block offending IP ranges and ASNs at the firewall level. Route traffic through Cloudflare's bot management. Deploy rate limiting. Use honeypots to detect and ban aggressive crawlers. Stop relying on robots.txt as your primary defense — it's a suggestion, not a shield.
Q: Is Amazon really doing something worse than other AI scrapers?
A: The honeypot evidence shows Amazonbot executing code from hidden shell scripts — that's qualitatively different from passive scraping. But the broader problem isn't Amazon-specific. The entire AI industry treats your website as a free data mine. Amazon just got caught being the most aggressive about it.