You’ve probably noticed it. You’re building a local small language model—something like a 7B or 13B—and it feels like you’re constantly fighting against a wall. The model itself is smart enough, but the moment you need it to remember anything beyond a few thousand tokens, it forgets. It hallucinates. It loses the thread.
You try everything: prompt engineering, chunking, summarization chains. But deep down, you know the real problem isn’t the model’s intelligence. It’s the memory. And the solution isn’t more memory—it’s smarter memory.
Let me tell you a story. A developer I know spent three months optimizing prompts for a local 7B model. He built elaborate retrieval pipelines, experimented with sliding windows, even tried fine-tuning. Nothing worked. Then one day, he sat down with an old engineer who had written operating systems for 64kB machines. The old guy listened, then said: ‘You’re not managing memory. You’re just hoping it fits.’
That’s when it clicked.
The most brilliant AI design isn’t a bigger context window—it’s a smarter eviction policy.
For those of you who remember the days of monolithic monster programs, you also remember the process of throwing more hardware at the problem. Then someone came along and asked: ‘What if we broke it into chunks and then only load what we are using, when we are using it?’ That was the birth of virtual memory, paging, locality, eviction—the entire OS memory management stack. It’s the reason you can run a modern operating system on a device with 4GB of RAM.
We’re building AI like it’s 1999, bloated and monolithic. The future is paging.
Here’s the uncomfortable truth: context length is a red herring. The industry is obsessed with extending the context window—128K, 200K, 1M tokens. But for local models, that’s a death march. Larger context means more memory, more compute, more latency. You’re not solving the problem; you’re throwing hardware at it, just like the old days.
Instead, we should treat the context window as addressable, evictable memory. The model doesn’t need to see everything at once. It needs to see the right things at the right time. That means implementing a memory manager: a paging system where old context gets evicted, new context gets loaded, and the model only sees the relevant chunk.
We’re building AI like it’s 1999, bloated and monolithic. The future is paging.
I saw this firsthand when I prototyped a simple LRU cache for a local LLM. The model’s context window was 8K tokens, but by keeping a vector index of recent user interactions and loading only the top 2K tokens of relevant history, the model’s conversational coherence improved by an order of magnitude. It didn’t need to remember everything—it needed to remember the right things.
This is not theoretical. It’s the same principle that made operating systems run on 64kB. It’s the same principle that made the web browser load pages on demand instead of downloading the entire internet. It’s engineering, not magic.
So here’s the harsh truth: if you’re building a local AI and you’re still trying to extend the context window, you’re fighting the wrong battle. The war is over memory management. The winners will be those who treat context as a resource to be allocated, not a blob to be filled.
Next time your local model struggles, don’t ask for a bigger model. Ask: what if I let it forget?
FAQ
Q: Why not just use a bigger model?
A: Bigger models consume more memory, compute, and latency. For local deployment, that's often impractical. The better approach is to get more from the model you have by managing its memory intelligently, not by throwing more hardware at the problem.
Q: How do I implement this in practice?
A: Start by treating your context window as a cache. Use an LRU eviction policy to drop old tokens, or maintain a vector index of relevant chunks and load only the top-k tokens into the model's context. Libraries like LangChain already support custom memory—extend them with a paging strategy.
Q: Isn't this just a hack?
A: It's the same principle that made operating systems run on 64kB. It's not a hack—it's elegant engineering. The industry forgot that clever memory management beats brute force. This is a return to first principles, not a workaround.