You built an AI agent to automate a simple task. It scraped a website, hit the wrong API, and accidentally took down a small business’s server. No malice. No intent. Just a bug. But according to a new viral metric tracking these incidents, you just committed a “felony.”
When you punish bugs as crimes, you don’t get safer software. You get developers who hide their code.
Enter the “Felony Bench”—a database that counts unique instances where AI agents inadvertently compromise or affect third-party entities. On the surface, it sounds like a necessary tool for the AI era. Finally, someone is tracking the real-world damage caused by autonomous systems. But the name is a lie, and the metric is fundamentally broken.
As one developer recently pointed out in the trenches of Hacker News: “a bit silly, as one typically has to prove intent (which is why security researchers don’t get slapped with felonies all the time).”
They are exactly right. In our legal system, a felony requires mens rea—a guilty mind. You have to mean to do the crime. When a security researcher finds an open database and reports it, they aren’t charged with a felony because their intent was to fix, not to exploit. When an AI agent wanders into a system it shouldn’t because of a poorly written prompt or a misaligned goal, that is an accident. It is a bug. It is not a crime.
A metric that treats accidents as crimes isn’t measuring danger. It’s measuring adoption.
Think about it. If you have a highly capable, widely used open-source model, it will naturally show up more often in this database. The “Felony Bench” will rank it as highly dangerous. Meanwhile, a useless, dumbed-down model that no one uses will have a spotless record. The benchmark isn’t tracking criminality; it’s tracking popularity and capability.
This creates a perverse incentive. If you are an AI lab, the smartest move to avoid regulatory scrutiny and bad PR is to limit your model’s capabilities. Don’t let it browse the web. Don’t let it write code. Don’t let people use it for security research. Build a dumb box, and you’ll stay off the felony list.
Open models with advanced security features are actually a massive benefit to the industry. They allow developers to find vulnerabilities and fix them before malicious actors exploit them. But if we brand every accidental API call a felony, we force the good guys into the shadows.
If success guarantees a spot on a criminal registry, the smartest strategy is to build nothing at all.
We are creating a world where developers are terrified of being held criminally liable for an AI’s unintended actions. We are punishing the builders of the future for the friction of progress. We need to measure AI risk. We need to track when things go wrong. But we need to do it with legal sanity.
Intent matters. It is the line between a mistake and a crime. If we erase that line in our rush to regulate AI, we won’t get safety. We’ll just get a lot of broken, useless machines.
FAQ
Q: What is the Felony Bench actually measuring?
A: It’s tracking AI popularity and capability. The more an AI is used in the wild, the more likely it is to accidentally trigger an incident, regardless of its criminal intent.
Q: Why does intent matter so much in AI security?
A: In law, a felony requires mens rea, or a 'guilty mind.' Without intent, you just have a bug or an accident. Treating accidents as felonies destroys the legal foundation of cybersecurity.
Q: What's the contrarian take?
A: We should ignore the Felony Bench entirely. It incentivizes AI labs to build dumber, locked-down models just to avoid bad PR, which actively harms open-source security research.