I Spent a Week Building an AI Knowledge Base for a Real Business. Here’s What Went Wrong.

It started with a confident slide into the client’s office. I had a demo ready. A knowledge base AI that could answer questions in seconds. I handed the laptop to the business lead and said, ‘Go ahead, ask anything.’

He asked about a speech from a leadership meeting. The AI fired back a perfect paragraph. But he didn’t look at the answer. He looked at the source citation. Three questions hit me: ‘What document is this from? What section? Is there an original title?’ Then he added: ‘Are there other versions of this speech from different meetings?’

I had nothing. The AI couldn’t tell him. That’s when I learned the hard truth: the most valuable work in an AI project has nothing to do with AI.

This was my week as a Forward Deployed Engineer (FDE). I built a knowledge base agent in ten minutes. It took me seven days to make it actually useful. And every single hour that mattered was spent on boring, non-technical work.

Let me show you what I mean.

The 10-Minute Illusion

You’ve probably seen the demos. Upload a PDF, ask a question, get an answer. Looks magical. Feels like AI is solving everything. But that’s a party trick, not a product.

I picked a real team—a department that handles internal reports, speeches, and research documents. Their pain point: scattered files, no way to find specific quotes fast. They wanted a search tool that gave them the exact text and told them where it came from.

Seems simple, right? I spun up an agent, connected a knowledge base, and in ten minutes I had a working prototype. I felt like a genius. Then I started the real work.

Step 1: Define What ‘Done’ Actually Means

Enterprise AI projects don’t fail because the tech is hard. They fail because nobody can agree on what ‘done’ looks like. One request for ‘search old documents’ becomes a full digital transformation with version control, permissions, and workflow automation.

I drew a hard line: this project only solves one problem—answering questions from a fixed set of finalized documents. No writing, no summarization, no PPT generation. Just search with source attribution. If you don’t know where the finish line is, you’ll never cross it.

I asked three questions before writing a single line of code: Who is this for? What data will we use? What counts as a success? Answer those, and you’ve already avoided 80% of the scope creep.

Step 2: The Data Grind

I received 50 documents. I didn’t upload them all. I spent hours inspecting each one. Format issues, conflicting versions, missing titles. The same speech appeared in three different reports with different context. Which one was the authoritative version?

I created a tracking sheet. Which documents needed cleaning? Which ones had overlapping content? Which had incomplete metadata? Most AI projects fail in the first 100 documents, because nobody bothers to check if the data is actually usable.

This is the part nobody demos. It’s boring. It’s tedious. And it’s the only thing that separates a toy from a tool.

Step 3: Test Like a Skeptic

I built the test questions. Direct queries, cross-document comparisons, time-bounded searches, and questions that should have no answer. If the AI couldn’t say ‘I don’t know’ when it didn’t have the data, it was useless.

Why test for ‘no answer’? Because in a business context, a hallucinated reply destroys trust. If a user asks for a 2023 revenue figure and the AI makes one up, that’s not a bug—it’s a liability. The best AI answer is often ‘I don’t have that information.’

I ran my internal tests. Everything looked good. I was confident. Then I brought in the real users.

Step 4: The Demo That Broke Me

Back to the moment I described at the start. The business lead didn’t care about the answer’s accuracy. He cared about provenance. He needed to know: who said this, when, in what context, and what other versions exist. The AI gave him text. He needed a story.

He asked for a specific speech from last year. The AI returned results from multiple years. He asked for a document type. The AI didn’t distinguish between a formal report and an internal memo. He asked for the original file name. The AI gave a truncated title. Business users don’t want a ‘semantically similar answer.’ They want the exact quote, from the exact document, used in the exact context.

That 40-minute meeting taught me more than a week of coding. The gap between a demo and a real deployment isn’t technical. It’s contextual.

Step 5: Classify the Feedback, Don’t Just React

After the demo, I had a list of complaints. But ‘the answer is wrong’ can mean a dozen different things. Missing data? Bad parsing? Wrong retrieval? Overconfident model?

I split the feedback into four buckets: data issues (missing or wrong files), parsing issues (missing metadata), retrieval issues (system didn’t find the right content), and reasoning issues (model made up an answer). Each bucket got a different fix. If you treat every complaint as a prompt-tuning problem, you’ll never fix the data.

Within a week, the client started asking for more features. They offered to supply more documents. They started talking about maintenance. That’s the real signal of success: when the user treats your prototype as part of their workflow.

What I Learned

The AI part took ten minutes. The rest took a week. And that week was all about human decisions: what to include, how to test, what to improve, how to communicate limitations.

Here’s my five-step framework that you can use tomorrow:

1. Write a one-page scope document: who, what, which data, what success looks like.
2. Curate a small but representative dataset. Verify every document’s format and metadata.
3. Build a test set that includes edge cases—especially questions that should have no answer.
4. Run a live test with real users. Watch them, don’t guide them. Record every frustration.
5. Classify each piece of feedback into data, parsing, retrieval, or reasoning. Fix each layer separately.

The real moat in enterprise AI isn’t the model. It’s the willingness to do the dirty work that models can’t do.

If you’re building an AI for a business, stop optimizing your prompts. Start cleaning your data. Start defining your boundaries. Start listening to the users who ask ‘where is this from?’

Because the future of AI isn’t about smarter models. It’s about smarter deployment. And that starts with the boring stuff.