You check your monthly API bill and feel a knot in your stomach. $10,000. $20,000. Maybe more. You’re not running a data center—you just built an agent that calls GPT-4 a few thousand times a day. The math doesn’t add up, but you keep paying because you need the quality.
What if I told you that you could cut that bill by 90% without changing your model, your prompts, or your application? Just by understanding two words: KV Cache and Prompt Cache.
The difference between a $10,000 API bill and a $1,000 one isn’t a better model—it’s how you structure your prompts.
Most developers don’t realize that LLM APIs have a hidden pricing loophole. When you send the same prefix—system prompt, tool definitions, output format—across multiple requests, the provider can reuse previously computed results. That reuse is called cache hit, and it costs as little as one-tenth of the standard input price. But here’s the kicker: your agent’s natural behavior—random timestamps, shuffled tool schemas, dynamic user IDs—silently destroys those cache hits, leaving you paying full price.
Let’s break down the two caches that matter.
KV Cache is the engine that makes text generation fast. When your model generates a token, it recalculates attention against every previous token. To avoid recomputing the same thing for every new token, the model stores the Key and Value vectors of each historical token. This happens inside a single request—you can’t control it, but it’s why long responses take more GPU memory.
Prompt Cache is the money-saver. It’s cross-request. If your agent sends the same 5,000-token system prompt across 50 requests, the API can compute the prefix once and reuse the result for the next 49. That’s where the 90% discount lives.
But here’s the catch: Prompt Cache is fragile. Add a timestamp at the beginning of your prompt? Cache miss. Change the order of your tool descriptions? Cache miss. Use a third-party API router that injects its own prefix? Cache miss—and you never see the discount.
Your third-party API router is silently robbing you. Here’s how.
Many developers fall for the lower headline price of routed APIs. But those routers often scatter your requests across different upstream instances—each with its own cache cold start. They may strip or modify cache control headers like Anthropic’s cache_control or OpenAI’s prompt_cache_key. Some even wrap your prompt with their own system message, breaking prefix matching entirely. The result: you pay the router’s low rate, but your effective cost per useful token skyrockets because you never hit cache.
So how do you actually get the 90% savings?
First, structure your prompts like a sandwich: static content on top, dynamic content on bottom. Your system prompt, tool definitions, output format, and project background go first—ideally in a fixed order and exact text. Never change them mid-session. Then append the variable parts: user question, retrieval results, tool outputs, timestamp. As long as the prefix is identical across calls, the cache will hit.
Second, stay on the official API or a provider that transparently passes cache metrics. If your dashboard doesn’t show cache creation tokens, cache read tokens, and a hit rate, you’re flying blind. Ask your provider: “Do you guarantee upstream persistence for cache? Do you forward cache control headers?” If they can’t answer yes, switch.
Third, batch similar tasks together with identical prefixes. Processing 20 files? Keep the same system prompt, only swap the file content at the end. Each call after the first will hit cache on the prefix.
Cache hits aren’t magic. They’re the direct consequence of prompt engineering discipline.
The biggest mistake I see: teams that optimize for prompt variety over structure. They rewrite system prompts for every use case, add dynamic elements like timestamps or random IDs at the top, and then wonder why their API costs don’t scale. The answer is that every change at the front of the prompt breaks the prefix, forcing the provider to recompute from scratch.
Real-world proof? In one of our client’s agent workflows, we rearranged a 4,000-token prefix to be fully static and moved the dynamic parts to the end. The cache hit rate went from 5% to 85% overnight. Their monthly bill dropped from $12,000 to $1,800—same model, same quality, same number of calls.
This isn’t a hack. It’s how the infrastructure was designed to work. The providers want you to hit cache—it saves them compute, so they reward you with lower prices. The problem is that most developers build their agents in a way that accidentally avoids caching. They’re not being punished; they’re just not cooperating with the system.
So here’s the final truth: If you’re not seeing cache hits on your API bills, you’re paying for stupidity, not intelligence. The model is already smart. Your job is to make its feed predictable. Start today: audit your prompt structure. Move the static parts up, freeze them, and watch your costs collapse.
FAQ
Q: Does caching actually save that much in practice?
A: Yes. We've seen agent workflows drop from $12,000 to $1,800 per month just by rearranging prompt order to maximize cache hits. The provider's pricing is explicitly designed to reward reuse—most teams just don't realize they're bypassing it.
Q: What's the practical implication?
A: Audit your prompts today: freeze system prompt, tool schemas, and output format into a fixed prefix. Append dynamic content (user query, retrieved docs) after that. Use official APIs that expose cache metrics. Avoid third-party routers that inject prefixes or scatter requests across regions.
Q: What's the contrarian take?
A: Third-party 'cheaper' API routers are often a trap. They break cache by routing to different upstream instances and stripping cache headers, making your effective cost higher than using the official API with cache optimization. The cheapest route is to cooperate with the provider's caching system, not bypass it.