A Cat Nearly Died Because of AI. Here’s What Every Developer Gets Wrong About RAG

It started with a phone call. A cat owner, worried, said only: “My cat is not normal. I want to check its heart.” The front desk at a small animal hospital scheduled a routine appointment. The cat arrived in critical condition. The hospital specialized in surgery, not cardiology. A near-death incident was set in motion — not by malice, but by a system that treated a vague description as a complete answer.

You’ve probably seen this pattern before. An AI product is built to answer questions faster. Data is dumped into a vector database, a chat interface is bolted on, and everyone claps. But in real-world, high-stakes verticals like healthcare, the most dangerous thing a RAG system can do is give a confident answer when the information is incomplete.

That cat’s story isn’t about search accuracy. It’s about governance. It’s about a system that had no protocol for what to ask, no way to escalate when the picture was blurry, and no mechanism to know when to stop and hand off to a human. And that’s the exact same failure that plagues most RAG implementations today.

We’ve been taught to treat retrieval-augmented generation as a search problem. Better embeddings, bigger context windows, fancier graph databases. But the cat’s case exposes the real bottleneck: RAG is not a model problem. It’s a workflow and governance problem. The moment you treat it as just another search optimization, you’re building a system that will confidently produce a dangerous answer — and call it a feature.

Let’s walk through the three breaks that turned a routine call into a crisis, and what they teach us about building AI that actually works in the real world.

Break 1: The Input Gap

The owner used street language: “not normal,” “check the heart.” The front desk had no approved list of questions to ask. No red flags, no escalation criteria. The system — the human system — relied entirely on personal experience. One person’s memory is not a protocol. If your RAG system only knows what it’s been told by a handful of people, it’s not a knowledge base. It’s a collection of lucky guesses.

An AI that transcribes the call is helpful. But an AI that pretends the original information is complete is dangerous. The first rule of medical RAG: preserve the uncertainty. Don’t let the model fill in missing symptoms from its general knowledge. That’s not a feature — that’s hallucination dressed up as helpfulness.

Break 2: The Knowledge Gap

No central repository of approved protocols, no capacity directory, no version-controlled questions. The knowledge lived in people’s heads — and it was inconsistent. Before you even think about chunk size or embedding models, you need to answer: What knowledge is authorized? Who reviewed it? When does it expire?

Most RAG tutorials skip this step. They start with data cleaning. But in a hospital — or any regulated environment — the first step is governance. You need a process that says: this document comes from Dr. X, reviewed by Dr. Y, valid until June 2026, and only applicable to these scenarios. Without that, your RAG system is just a faster way to spread wrong information.

Break 3: The System Gap

Patient records in one SaaS, X-rays in another, lab reports in a filing cabinet. The doctor couldn’t access records from home. The front desk had to physically search for paper files. Sound familiar? RAG can’t fix broken data infrastructure. It can only amplify its flaws.

Before you connect a language model to your data sources, make sure those sources are actually accessible, deduplicated, and permissioned. Otherwise, your RAG will be a beautiful interface to a chaotic mess — and the human will still be the one to blame when things go wrong.

What Actually Works: Three Retrieval Paths, Not One

Once you’ve fixed the governance, you can design the RAG. But here’s the twist: most RAG systems treat all knowledge as equal. In reality, you need three separate retrieval paths.

Institutional knowledge (protocols, drug lists, capacity) — use hybrid keyword + semantic search with metadata filters. Patient data — first confirm identity, then retrieve by structured fields (patient ID, date) before using vector similarity. External medical evidence — separate index, stricter permissions, and citations that link every claim to a specific version.

Mixing them into one pool is a recipe for “the model said so” — a phrase that terrifies any accountable professional.

When to Say No: The Ultimate Test of a RAG System

Here’s the uncomfortable truth that no vendor will tell you: the best RAG system is the one that knows when to stop answering. When the required information is missing, when the patient identity is uncertain, when the source material is out of date — the system should refuse to answer, escalate to a human, and explain why.

That’s what the cat’s hospital needed. A system that, when the owner said “not normal,” would trigger a structured inquiry, flag the missing data, and if the risk threshold was met, route the call to a nurse — not just schedule an appointment.

Most AI product managers treat the “I don’t know” response as a failure. But in high-stakes applications, it’s the only responsible answer. Building a system that confidently says “this is what the cat has” is easy. Building one that says “I don’t have enough information, let me hand you to a doctor” is the real engineering challenge.

So before you upgrade your embedding model or start implementing GraphRAG, ask yourself: have you mapped out the workflow? Do you know who owns each piece of knowledge? Have you defined what “stop” looks like? If not, you’re not building a RAG system. You’re building a catastrophic liability.

That cat survived. But the next one might not. The AI that saved it wasn’t smarter — it was more careful. And that’s a lesson every developer needs to learn.