Skip to content

IWENAI

Ideas Weave Every Narrative with AI.

Home › AI & Machine Learning › I Built an AI That Reviews PRs by Watching Them Work. Here’s the Brutal Truth About False Positives.

I Built an AI That Reviews PRs by Watching Them Work. Here’s the Brutal Truth About False Positives.

📅 August 18, 2026 📂 AI & Machine Learning

You know the feeling. You’ve got five sessions of Claude Code and Cursor running on different worktrees, shipping code like a factory. The only way to know if something broke is when it hits production and a user sends a screenshot of an orange page. Your PR review skills catch code-level errors, but they’re blind to UI regressions. That’s a nightmare.

I got tired of it. So I built Kery — a tool that opens your PR in a real browser, logs in, and records a video of the feature working or not. The comment is the product. I wanted the artifact to be evidence, not an opinion. It reduces the cognitive load from your brain needing to process every PR for what it is. You watch what happened instead of reading a claim about it.

But here’s the twist: the tool’s biggest challenge isn’t technical. It’s psychological.

Kery reads your PR diff, understands the change intent, opens your preview deployment, logs in (past Clerk, Supabase, Auth0, even 2FA and Vercel preview protection), and then tries to perform exactly what the PR is for. It drives off the accessibility tree instead of CSS selectors, so it survives DOM churn. It gets quieter every run — a triage pass dedupes findings and remembers the ones you called false positives, with confidence decay so old memory doesn’t compound.

On my benchmark, it catches about 60% of planted bugs. It finds real bugs too. But it also invents about two false positives per run on a clean page. False positives are the reason tools like this get switched off in two weeks. That’s the number that keeps me up at night.

Most observers focus on the technical challenge of agent-based browser testing. The real hurdle is psychological. Every false positive chips away at the user’s trust. The tool’s long-term survival depends not on catching more bugs but on managing the noise-to-signal ratio better than human intuition. If Kery tells you something is broken and it isn’t, you start ignoring it. The boy who cried wolf isn’t a technical problem; it’s a trust problem.

I’ve open-sourced the engine under Apache-2.0. You can run it locally with Docker — npx keryai sets up a dashboard on localhost:11111, your own keys, runs against any web app. Or use the cloud version with a GitHub App. Either way, I want you to break it. Nobody except me has run this against a repo I don’t own, so I’d rather hear that it fell over than that it’s neat.

If you try it and get a false positive, reply with the details. That’s more useful than a compliment. The tool’s future depends on the community’s willingness to report the noise. Because the promise is huge: proof, not opinions. But the price of admission is trust. And trust is earned, one false positive at a time.

FAQ

Q: What about false positives? Doesn't that make the tool useless?

A: False positives are the biggest risk. The creator is open about it – currently ~2 per run. If you have a low tolerance for noise, wait for improvements. But if you're drowning in manual PR reviews, the trade-off might be worth it, especially if you help tune the engine by reporting false positives.

Q: How do I use this in my workflow?

A: You can run the open-source engine locally with Docker (npx keryai) or use the cloud version with a GitHub App. It reads your PR diff, opens the preview deployment, logs in, and records a video. No test scripts needed. The output is a comment with video evidence. Use it for UI/UX regressions after every PR.

Q: Isn't this just another AI tool that will be forgotten in a month?

A: The hot take: most AI tools are solutions in search of a problem. Kery solves a real pain – the cognitive load of manual PR review. But its survival depends on trust. The open-source engine and the creator's brutal honesty about false positives give it a fighting chance. If the community helps reduce false positives, it could become indispensable.

Accessibility Account Security Action Adoption AI Open Source
📎 Source: View Source

📖 Related Articles

AI Detectors Don’t Catch Cheaters. They Punish the Innocent.

Imagine pouring six months of your life into a dissertation. You survive on cold coffee,…

Your BMW Is Now a Relic: The German Auto Industry’s Collapse Is Reshaping Europe’s Soul

You've probably noticed the headlines. BMW cutting jobs. Volkswagen closing plants. Mercedes warning of a…

The Microsoft Interview Question That Made Me Fail — and Why I’d Do It Again

You know that moment in an interview when your brain freezes, and you feel the…

Your AI Assistant is Holding You Hostage to Your Past Self

Six months ago, you told ChatGPT you were on a diet. You wanted low-calorie meal…

← The EU's Age Verification Plan Is a Digital Gatekeeper. One Developer Just Proved There's a Better Way. Your Robot Update Strategy Is Doomed. It's Not the Code. →

© 2026 IWENAI. Ideas Weave Every Narrative with AI.

JSON Feed RSS API Sitemap