You’re paying premium prices for an AI that spends almost all its time re-reading its own homework.
I don’t mean that metaphorically. I mean it literally. Someone aggregated 32 Claude Code sessions and discovered that 96.8% of all tokens consumed went to re-reading previous context. Less than 4% went to actually generating new, useful output.
Think about that for a second. You ask Claude to write a function. It writes it. Then you ask a follow-up. To answer that follow-up, it has to re-ingest everything it already told you, plus everything you discussed, plus the system prompt, plus the tool definitions. Every single turn. Every single time.
You’re not paying for intelligence. You’re paying for an AI to re-read its own diary before every sentence it speaks.
Here’s where it gets worse. The longer your conversation, the more expensive each response becomes — not linearly, but exponentially. A 50-turn coding session doesn’t cost 50x a single turn. It costs dramatically more, because turn 50 requires re-reading all 49 previous turns before generating a single new token.
This isn’t a bug. It’s the fundamental architecture of transformer models. They have no persistent memory. No internal state that carries forward. Every time you send a message, the model starts from scratch and has to reconstruct the entire conversation from text. It’s like waking up every morning with amnesia and having to read your own journal just to remember who you talked to yesterday.
The industry’s response to this? Bigger context windows. 100K tokens! 200K! 1 million! The marketing screams about how much the model can “remember.” But nobody’s talking about the cost of that remembering.
The obsession with longer context windows isn’t solving the memory problem — it’s amplifying the billing problem.
Every additional token of context you stuff into a window is a token the model has to process again and again on every single subsequent turn. A 100K context window doesn’t mean the model holds 100K tokens in elegant, efficient memory. It means the model re-reads 100K tokens before every response. You’re being charged for the re-reading, not the remembering.
I’ve seen this firsthand in my own Claude Code sessions. A complex refactoring task that spans 30 turns ends up consuming hundreds of thousands of tokens — not because the work is complex, but because the model is re-processing the entire conversation history on turn 31. The actual code generation? A rounding error in the token budget.
And here’s the twist nobody wants to hear: this isn’t something a software update fixes. It’s baked into the transformer architecture itself. The attention mechanism that makes LLMs powerful requires re-attending to all previous tokens on every forward pass. There’s no shortcut. No cache that makes it free. The re-reading IS the computation.
Every transformer is a goldfish with a photographic memory — it remembers everything, but only by constantly re-experiencing it.
This has real consequences for anyone building with or paying for AI coding tools. Your costs don’t scale with the complexity of the task. They scale with the length of the conversation. A simple question asked at turn 1 costs pennies. That same question asked at turn 40 costs dollars. Same question. Same answer. Ten times the price.
The practical implication? Start fresh sessions aggressively. Don’t let conversations sprawl. Break complex tasks into focused, short interactions. Every token of history you carry forward is a tax on every future response.
And for the industry? Stop celebrating bigger context windows like they’re a feature. They’re a cost multiplier dressed up as a capability. The real breakthrough won’t be a model that can read a million tokens. It’ll be a model that doesn’t have to re-read any of them.
The next leap in AI won’t come from giving models more to remember. It’ll come from freeing them from the burden of remembering at all.
Until then, check your token usage. You might be horrified to discover how much of your budget is going to an AI whispering its own history back to itself before it can answer your question.
FAQ
Q: Isn't KV caching supposed to solve this re-reading problem?
A: KV caching helps with latency on individual turns, but it doesn't eliminate the fundamental cost. The cache still has to be stored, loaded, and attended over. And in multi-turn agentic sessions like Claude Code, cache invalidation happens frequently as context changes. It reduces the penalty but doesn't remove the architectural tax.
Q: So should I just avoid long conversations with AI entirely?
A: Not entirely, but be strategic. Break complex tasks into focused sessions. Start fresh when context is no longer relevant. The cost difference between a 5-turn session and a 50-turn session isn't 10x — it's often 50x or more, because every turn re-processes all prior turns. Treat conversation length as a cost variable, not just a convenience.
Q: Doesn't this mean the entire long-context trend is a scam?
A: Not a scam, but deeply oversold. Long context windows are genuinely useful for single-pass tasks like analyzing a large document. The problem is multi-turn usage, where the cost compounds. The industry markets context length as 'memory' when it's really 're-processing capacity.' That distinction matters enormously for your bill.