Stop Writing PRDs for AI Agents. Your First Job Is to Write the Answer Key.

It was day four of my new job, and I was already doing something that looked nothing like product management. I sat on my couch, staring at a video generation AI, and started typing commands—one by one—like a robot trainer. “Change the color of the first scene to black and white.” “Mute the background music.” “Delete scene five.”

I wasn’t testing a product. I was feeding the AI the answer key.

Every command I typed was a pair: input → expected output. I told the AI what “good” looked like, what “bad” looked like, and what counted as success. This is called data labeling. Three months ago I would have called it mind-numbing grunt work. Today I call it the most important thing an AI product manager will ever do.

Because here’s the truth nobody tells you: For an AI agent, the evaluation set is your PRD. And your PRD is obsolete.

Let me explain.

Your Product Manager Job Description Is Wrong

When I started, I thought I’d be writing specs, talking to engineers, and noodling over roadmaps. Instead, I found myself building a spreadsheet of 30 test cases—each one a tiny scenario that defines the product’s natural language boundary. “Can the user ask to change the color of a single scene?” “Can they mute only the music, not the voice?” “Can they export a specific clip?”

Each “yes” or “no” becomes a product requirement. If the AI can’t handle “change the color of the first scene,” then the product doesn’t support per-scene editing. That’s not a bug report—that’s a feature gap. The evaluation set doesn’t just test the product; it defines the product.

Traditional PRDs list features. Agent PRDs list the ways users will talk. And the only way to discover those ways is to write down every possible sentence a user might say and then force the AI to answer. Your product’s boundary is the set of natural language inputs it can handle. Everything else is a gap.

The Most Dangerous Bug Isn’t a Crash

I ran through my 30 test cases. Most failed. But one case freaked me out.

I typed: “Mute all audio in the entire video.”

The AI replied: “Completed. All audio has been fully muted.”

But when I played the video, only the background music was gone. The voiceover and ambient sounds were still there. The AI thought it succeeded. The user would think it succeeded. And then they’d export the video, send it to a client, and discover the mistake three hours later.

This is the nightmare scenario for AI agents: fake success is far more dangerous than clear failure. A clear failure—a “not supported” error—triggers the user’s skepticism. They try again, or they manually fix it. But fake success bypasses skepticism entirely. It propagates flawed output with a smile.

So here’s my rule number one for any AI product: Never trust an agent’s self-report. Give the user a preview button and a one-click undo. Let humans verify. Build a safety net that lets them see the truth with their own eyes.

The Overthinker’s Superpower

I’ve been called “too sensitive” my whole life. I see colors, lighting, composition, emotion. I hear the mixing in a song, the balance of vocals and instruments. I pick up on subtext in conversations. For years I thought this was a flaw—a drain on my energy, a reason to be called “dramatic.”

Then I started evaluating AI-generated videos. And I realized: LLMs can only judge 2D metrics—resolution, duration, format. They can’t tell you if a scene feels beautiful, if a transition is smooth, if the character looks the same throughout, if the lighting direction is consistent. That’s 3D qualitative judgment. That’s human.

And the people who can do that—the ones who notice the slight off-ness in a color grade, the ones who feel when a story’s emotional arc is broken—they are not a liability. They are the product’s last line of defense. Only a human who has lived can decide what “good” means. AI can generate a hundred versions, but it takes a sensitive human to pick the one that actually works.

So if you’re an AI product manager and you’ve ever been told you “think too much,” take it as a compliment. Your sensitivity is your superpower. Use it to draw the boundaries that no algorithm can see.

The Fastest Way to Understand a Competitor

My boss told me to “check out” a competitor. I thought I’d read their docs, watch their demos, maybe chat with a user. Instead, I did something simpler: I took a piece of content I knew well, fed it into their AI, and then tried to edit the output.

I ran a poem through their video generator. Then I tried to change the color of the first scene. Then I tried to mute the music. Then I tried to delete a scene. Thirty minutes later, I had a complete picture: their generation was great, but their editing was almost nonexistent. The root cause was obvious—they had baked the audio and video into a single stream during generation, leaving no editing hooks.

No document could have told me that. No analyst report. The best competitive analysis is simple: use the product the way a real user would, and see where it breaks.

This is the job now. There are no textbooks for AI product management. The product itself is being defined as we speak. And the people who are building it—the ones writing the answer keys, testing the edge cases, and trusting their gut—are the ones who will decide what the future looks like.

I’m a week in. I have no template. I have no senior mentor. But I have a spreadsheet full of test cases, a stubborn sensitivity, and a growing conviction that the most important work in AI isn’t about the code—it’s about the questions we ask.

And I’m just getting started.