You’ve probably felt it. That creeping anxiety that the playbook you’ve used for years to build products is suddenly slipping through your fingers. YC President Gary Tan recently said that if you’re willing to spend $100,000 a year on tokens right now, you’re essentially living in 2028. That’s not just a catchy line; it’s a death knell for traditional product management.
When Dianne Penn, the first technical Product Manager at Anthropic, talked about this on Lenny’s Podcast, it crystallized something we’re all feeling. As token costs plummet and model capabilities leap forward in unpredictable, discontinuous jumps, the scarce resource has fundamentally shifted. Execution is cheap. When execution costs approach zero, the only scarce resource left is judgment.
For decades, a PM proved their worth by writing a pristine Product Requirements Document (PRD). You defined the vision, mapped the features, and handed it off to engineering. But if you’re building on top of an AI model—a system that is inherently unpredictable and outputs something different every time—a static document is practically useless. You can’t spec your way out of hallucinations.
This is why Penn’s team at Anthropic operates on a radically different principle: Evals are the new PRDs.
Consider the early days of Claude 2. The team kept getting vague feedback that ‘Claude doesn’t follow instructions.’ In the old world, a PM would write a PRD demanding ‘better instruction following.’ But Penn’s team dug into the raw conversation logs. They discovered that 80% of these complaints were actually about one thing: Claude couldn’t reliably output correct JSON format. So, they gathered 30 to 40 specific examples of failures and turned them into a pass/fail test. That was their eval.
You cannot improve what you cannot measure, and an AI model is the ultimate unmeasurable beast. Evals aren’t a trendy new framework; they are the only bridge between chaotic model behavior and actual user value. It’s test-driven development for product managers. If you can’t define the eval, you don’t actually understand the problem.
But this shift requires a massive change in how we operate. It requires the kind of agility that Anthropic showed when they launched the ‘Golden Gate Claude’ experiment. They noticed a quirky feature where the model obsessed over the Golden Gate Bridge, and within 24 hours, four teams rallied to push it live to users. It didn’t make money. It didn’t scale. But it proved that the organization could move at the speed of discovery.
True bottom-up innovation isn’t a slogan on a wall; it’s the willingness to drop everything for 24 hours to put a fascinating quirk in front of users. That kind of speed is terrifying to traditional PMs who are used to gating releases behind months of planning.
But here is the twist that most people miss when they declare the PRD dead: PRDs are still vital. They just serve a different master now.
Penn is clear that Anthropic still writes PRDs for every model generation. Why? Because when you are exploring a vague, unproven opportunity—like giving an AI the ability to control a computer screen—you don’t have concrete user pain points yet. You have a vision. The PRD hasn’t died; it just changed jobs. It’s no longer for measuring progress, it’s for aligning armies.
If the problem is crystal clear and needs iterative measurement, you use evals. If the problem is murky and you need to convince a massive team of engineers, lawyers, and safety experts to walk in the same direction, you write a PRD. Choosing between them shouldn’t be a matter of habit; it should be dictated by the clarity of the problem in front of you.
This brings us back to judgment. As AI models get smarter, they will write the code, they will design the UI, and they will even draft the PRDs. What they cannot do is decide what is actually worth building. That requires human taste, grit, and the intuition to know when to push back.
Anthropic designed Claude to literally argue with you. They believe an AI that just agrees with you is useless; a truly valuable partner pushes back at critical moments. The same is true for product leaders. The most valuable collaborators aren’t the ones who blindly execute your vision; they are the ones willing to push back when your vision is flawed.
OpenAI and Anthropic are locked in a fierce rivalry, yet leaders from both camps are arriving at the exact same conclusion: implementation is no longer the bottleneck. Judgment is. If you are still clinging to traditional, document-heavy, linear planning, you aren’t managing a product. You are managing your own irrelevance. Stop polishing your PRDs and start defining your evals. The models are waiting for you to tell them what actually matters.
FAQ
Q: If AI models are getting so smart and unpredictable, why do we need rigid evals instead of just letting the model figure it out?
A: Because unpredictability is exactly why you need evals. You can't steer a car blindfolded. Evals act as your standardized feedback loop, telling you if a model's 'improvement' actually solved the user's specific pain point or just introduced a new, fancy failure. Without them, you're just guessing.
Q: If evals are the new PRD, should I stop writing product requirement documents entirely?
A: No. PRDs and evals serve different stages of problem clarity. Use a PRD when the opportunity is vague and you need to align large, cross-functional teams on a shared vision. Use evals when the problem is specific, measurable, and requires iterative testing against model behavior.
Q: Isn't this just a fancy way of saying PMs need to do more testing?
A: It's the opposite of traditional QA testing. Traditional testing checks if the code matches the spec. Evals define what the spec should even be in the first place, because with AI, you often don't know what the model is capable of until you run the eval. It's product discovery disguised as measurement.