Skip to content

IWENAI

Ideas Weave Every Narrative with AI.

Home › AI & Machine Learning › I Built an AI That Reviews PRs by Watching Them Work. Here’s the Brutal Truth About False Positives.

I Built an AI That Reviews PRs by Watching Them Work. Here’s the Brutal Truth About False Positives.

📅 August 18, 2026 📂 AI & Machine Learning

You know the feeling. You’ve got five sessions of Claude Code and Cursor running on different worktrees, shipping code like a factory. The only way to know if something broke is when it hits production and a user sends a screenshot of an orange page. Your PR review skills catch code-level errors, but they’re blind to UI regressions. That’s a nightmare.

I got tired of it. So I built Kery — a tool that opens your PR in a real browser, logs in, and records a video of the feature working or not. The comment is the product. I wanted the artifact to be evidence, not an opinion. It reduces the cognitive load from your brain needing to process every PR for what it is. You watch what happened instead of reading a claim about it.

But here’s the twist: the tool’s biggest challenge isn’t technical. It’s psychological.

Kery reads your PR diff, understands the change intent, opens your preview deployment, logs in (past Clerk, Supabase, Auth0, even 2FA and Vercel preview protection), and then tries to perform exactly what the PR is for. It drives off the accessibility tree instead of CSS selectors, so it survives DOM churn. It gets quieter every run — a triage pass dedupes findings and remembers the ones you called false positives, with confidence decay so old memory doesn’t compound.

On my benchmark, it catches about 60% of planted bugs. It finds real bugs too. But it also invents about two false positives per run on a clean page. False positives are the reason tools like this get switched off in two weeks. That’s the number that keeps me up at night.

Most observers focus on the technical challenge of agent-based browser testing. The real hurdle is psychological. Every false positive chips away at the user’s trust. The tool’s long-term survival depends not on catching more bugs but on managing the noise-to-signal ratio better than human intuition. If Kery tells you something is broken and it isn’t, you start ignoring it. The boy who cried wolf isn’t a technical problem; it’s a trust problem.

I’ve open-sourced the engine under Apache-2.0. You can run it locally with Docker — npx keryai sets up a dashboard on localhost:11111, your own keys, runs against any web app. Or use the cloud version with a GitHub App. Either way, I want you to break it. Nobody except me has run this against a repo I don’t own, so I’d rather hear that it fell over than that it’s neat.

If you try it and get a false positive, reply with the details. That’s more useful than a compliment. The tool’s future depends on the community’s willingness to report the noise. Because the promise is huge: proof, not opinions. But the price of admission is trust. And trust is earned, one false positive at a time.

FAQ

Q: What about false positives? Doesn't that make the tool useless?

A: False positives are the biggest risk. The creator is open about it – currently ~2 per run. If you have a low tolerance for noise, wait for improvements. But if you're drowning in manual PR reviews, the trade-off might be worth it, especially if you help tune the engine by reporting false positives.

Q: How do I use this in my workflow?

A: You can run the open-source engine locally with Docker (npx keryai) or use the cloud version with a GitHub App. It reads your PR diff, opens the preview deployment, logs in, and records a video. No test scripts needed. The output is a comment with video evidence. Use it for UI/UX regressions after every PR.

Q: Isn't this just another AI tool that will be forgotten in a month?

A: The hot take: most AI tools are solutions in search of a problem. Kery solves a real pain – the cognitive load of manual PR review. But its survival depends on trust. The open-source engine and the creator's brutal honesty about false positives give it a fighting chance. If the community helps reduce false positives, it could become indispensable.

Accessibility Account Security Action Adoption AI Open Source
📎 Source: View Source

📖 Related Articles

Your AI Agent Is Bleeding Tokens. Stop Treating It Like a Chatbot.

You’ve felt it. That agonizing pause. You ask your AI agent to execute a complex…

Stop Hand-Coding Web Animations. The Bottleneck Is Now Your Imagination.

We've all been there. You have a brilliant vision for a web interaction—a cascade of…

Stop Calling T. Rex a Reptile. It Was Basically a Giant, Murderous Bird.

You’ve probably been lied to about the T. rex your entire life. We all grew…

The Chrome Web Store Just Proved You Don’t Own Your Tools

You wake up, brew your coffee, open your browser, and reach for that little icon…

← The FBI Wasn't Hacked by a Superpower. It Was Hacked by an HR Portal. Stop Blaming Your Engineers for SAML. The Protocol Was Already Dead. →

© 2026 IWENAI. Ideas Weave Every Narrative with AI.

JSON Feed RSS API Sitemap