A developer posted on Hacker News that they were heading out to sea for a week — no cloud, no API calls, no lifeline — with a MacBook Pro M2 Max packing 96GB of unified memory. Their mission? Rebuild a customized Shopify store, clean up SKU metadata across an entire product catalog, and build a utility script for their photography workflow. All offline. All local.
And almost immediately, the comments did what comments always do: they recommended the biggest models that would fit in memory. Llama 2 13B. Mixtral. The works. Because the instinct of every developer who discovers local LLMs is the same: if the memory is there, fill it.
That instinct is wrong. And it’s killing your inference speed.
Here’s the paradox nobody talks about. The M2 Max has 96GB of unified memory — a staggering amount by any standard. But unified memory is not the same as GPU memory. The GPU can access it, yes, but the bandwidth and the way memory gets carved up for the neural engine versus the GPU versus the CPU means that loading a 13B parameter model doesn’t just eat RAM — it creates fragmentation, it slows token generation, and it makes batch processing feel like wading through wet concrete.
Having 96GB of memory doesn’t mean you should fill it with a model that thinks like a distracted encyclopedist.
The real play — the one that actually works when you’re 200 miles offshore with a satellite phone and a deadline — is a smaller, fine-tuned model. Think 7B parameters. Think Q4 quantization. Think a model that doesn’t know everything about everything but knows exactly what you need it to know: how to write clean code, how to extract structured data from messy product descriptions, how to generate the JSON you need for a Shopify metafields import.
A 7B model with 4-bit quantization loads fast, runs cool, and generates tokens at a pace that feels like a conversation, not a waiting room. When you’re batch-processing 2,000 SKUs, that difference isn’t marginal. It’s the difference between finishing before dinner and finishing before you lose signal entirely.
A 7B model that knows exactly what it’s doing will always beat a 13B model that’s still figuring it out.
This is the lesson that the local LLM community keeps relearning the hard way. Bigger models are seductive. They feel like power. But power without precision is just noise, and noise doesn’t clean up your product catalog. What you need is a model fine-tuned for code generation and structured data extraction — something like a CodeLlama 7B variant or a Mistral 7B instruct model with the right system prompt — running through a framework like llama.cpp or Ollama that handles memory efficiently on Apple Silicon.
The Shopify rebuild? That’s code generation. You want a model trained on code, not a generalist that hallucinates Liquid template syntax. The SKU metadata cleanup? That’s structured extraction — give the model a schema, feed it messy product descriptions, and let it output clean JSON. The photography workflow utility? That’s a script, not a dissertation. None of these tasks require a model that can also write poetry in Latin.
And here’s where it gets interesting. When you strip away the cloud, something shifts in how you work. You stop asking the model to do everything and start engineering around its strengths. You write better prompts because you can’t afford bad outputs. You build pipelines because you can’t babysit every response. You become, paradoxically, a better engineer — because the crutch is gone.
The cloud didn’t make you smarter. It made you dependent. Offline is where you find out what you actually know.
The developer heading out to sea with their M2 Max isn’t just running a local LLM. They’re running an experiment in self-reliance. And the answer isn’t the biggest model they can cram into memory. It’s the smallest model that does the job precisely, repeatedly, and fast enough to matter.
Because when you’re offline, latency isn’t an inconvenience. It’s the clock. And the clock doesn’t care how many parameters your model has.
So before you download that 13B model because your machine ‘can handle it,’ ask yourself: do you need a model that knows everything, or a model that does the one thing you need — fast, clean, and without a data center in sight?
The best local LLM isn’t the one that fits in your memory. It’s the one that fits your problem.
FAQ
Q: Isn't a 13B model objectively smarter than a 7B model?
A: On a general benchmark, maybe. But you're not running benchmarks — you're cleaning SKU data and writing Liquid templates. A 7B model fine-tuned for code will outperform a 13B generalist on specific tasks while running 2-3x faster. Smart is relative. Fast and correct is absolute.
Q: What does this mean for developers building offline AI workflows?
A: Stop optimizing for model size. Start optimizing for task fit, quantization strategy, and inference latency. Pick the smallest model that reliably handles your use case, quantize it to Q4, and build a pipeline around it. Your memory bandwidth will thank you.
Q: Is the local LLM movement just for people without internet access?
A: No. Going offline is a forcing function that makes you a better engineer. When you can't fall back on GPT-4, you write tighter prompts, build real pipelines, and choose models deliberately. The cloud is a crutch. Local AI is where you actually learn what the technology can do.