Stop Building AI Agents Until You’ve Done These 6 Things

I’ve seen it happen more times than I can count. A company spends six figures on an AI platform. The sales demo was dazzling. The tool promised to answer any question, automate workflows, and turn years of documents into a genius assistant. Three months later, the agent is answering with garbage. Nobody trusts it. The project goes into a drawer labeled “lesson learned.”

The problem wasn’t the AI. It was what they didn’t do before they bought it.

I’m an FDE — a Field Data Engineer. My job is to walk into a company, sit with the people who actually do the work, and figure out what knowledge is real and what is just noise. Last week I spent five days embedded with a large construction services firm in Wuxi, China. They wanted an AI agent for their engineers. They had documents, drawings, spreadsheets, and decades of experience locked inside senior experts’ brains.

And they were two weeks away from buying a tool we all knew would fail.

The most dangerous thing in enterprise AI is a confident agent trained on garbage. That’s the line I use to wake executives up. And it works. Because by the end of that week, we didn’t touch a single AI platform. We spent every minute doing something far more valuable: a knowledge audit.

Step 1: Find Where Your Knowledge Actually Lives

You’d be surprised how many companies can’t answer this. “We have a shared drive.” “It’s on the intranet.” “The senior engineers just know.” That’s not an answer — that’s a risk map of failure.

I sat with the IT team first. We mapped every system: online databases, shared network folders, personal hard drives, WeChat groups, even the sticky notes on desks. We asked: Who owns this? When was it last updated? Is there a version history?

Your first batch of knowledge can’t be everything. It has to be traceable, assignable to a person, and testable. If a document’s source is unknown, it’s poison for your AI.

AI can’t fix messy data; it only amplifies it. That quote gets a lot of screenshots because it’s exactly what happens when you skip this step.

Step 2: Be Brutal About What Goes In First

Every department will tell you their files are critical. They’re lying. Not intentionally — but most knowledge has too many caveats to be useful in a first release.

We interviewed a road-bridge engineer. He told me: “Our progress schedules are theoretically knowledge. But we build water treatment plants and highways. The methods are completely different. You can’t transplant.” He was right. Forcing that into an agent would create a tool that spits out context-free answers that look correct but are fundamentally wrong.

Your first knowledge domain should score high on business value, frequency of questions, and — crucially — expert willingness to review answers. If an expert isn’t ready to validate the QA, don’t include it yet.

Step 3: Don’t Talk to Just the Boss

The executives commissioned the project. They want “organizational capability” and “digital transformation.” That’s fine, but the agent has to work for the guy in the back office who needs to find the latest code update before lunch.

I sat with junior staff, project managers, and administrative assistants. One told me a painful truth: “I find the file, but the content is useless. It only shows the design — not what actually got built. I have to tell every requester: this is not as-built data.”

That’s a knowledge boundary that can’t be ignored. Your agent must know not just what it has, but what it doesn’t have. It needs to flag uncertainty, not hide it.

An AI that doesn’t know its own limits is a liability, not an asset.

Step 4: Extract the Expert’s Judgment, Not Just Their Rules

The most valuable knowledge is unwritten. It’s the engineer who says, “In this situation, you ignore the standard spec because the soil here is unstable.” That judgment call is worth a hundred manuals.

During interviews, I don’t ask “What documents are useful?” I ask “What do people ask you every week? How do you decide the answer? What would you tell them to watch out for?”

That’s how you build a knowledge structure that includes scenarios, conditions, steps, and risk warnings. Without that, your agent will be little more than a glorified search bar.

Step 5: Collect Real Questions During the Audit

Don’t wait until after the agent is built to test it. Start testing the moment you step in. Every question an employee asks an expert is a potential test case. Every frustration with finding data is a requirement.

We gathered 40 actual questions from interviews: “What’s the maximum height for this zone?” “Which fire safety code applies to a mixed-use building?” “Can I reuse this design for a similar project?” Each one became a measurement of success.

Your first knowledge base is not a fully baked agent — it’s a prototype that proves the pipeline works. If it can answer those 40 questions with proper citations, you’ve got a foundation. Expand from there.

Step 6: Set the Boundaries Before the First Demo

Enterprise AI projects die because of mismatched expectations. The client thinks the agent will automate complex decision-making. The team knows it can only retrieve and summarize. That gap kills trust.

In the first week, I make the boundaries explicit. What can the agent do? Retrieve, reorganize, flag, and suggest. What should it never do? Make a judgment call that requires professional responsibility, sign off on a legal requirement, or override an engineer’s expertise.

AI amplifies human competence only when humans stay responsible. That boundary is non-negotiable.

By day five, we had a roadmap. No tool. No contracts. Just six steps that would make any AI investment actually work. The client was relieved. They had found the real bottleneck — and it wasn’t the algorithm.

The next time someone pitches you a flashy AI agent, do yourself a favor. Ask one question: “Have you done the knowledge audit yet?” If they say no, say thank you and walk away.