You’ve seen the benchmarks. Laguna S 2.1 on an M3 Ultra: 52 tokens per second, 38.5 GB peak memory. Impressive, right? But if you’ve actually tried to run these models locally, you know the truth: the numbers are a beautiful lie.
The fastest inference engine is the one that doesn’t crash.
I spent the last week trying to replicate that feat. I have the hardware. I have the patience. But what I found is a story that no benchmark will tell you: the real bottleneck isn’t compute—it’s memory bandwidth, and more crucially, the sheer fragility of the software stack.
One developer on the GitHub thread put it bluntly: “I’ve got the nvfp4 running on a spark, impressed enough so far I’m going to see if it can replace my qwen3.6 27B usage. Have had issues with the dflash drafter and vllm, crashes, OOMs, broken thinking… so I’m back to the mid teens tok/s.”
That’s the voice of reality. The promise of local AI—privacy, zero latency, no API costs—is real. But the path is littered with OOMs, segfaults, and broken thinking traces. The reason isn’t that Apple Silicon is slow. It’s that the memory bandwidth of unified memory is a hard wall, and the frameworks to exploit it are still in beta.
You can’t benchmark your way around a memory bandwidth wall.
Most people focus on model size or token speed. They ask: “Can I run a 70B model?” The better question is: “Can I run it without crashing after 10 minutes of conversation?” The answer today is: maybe—if you have the right quant, the right drafter, and a lot of luck.
Here’s what the benchmarks don’t show: the 52 tok/s number comes from a specific setup—nvfp4 quantization, a draft model, and a carefully tuned inference engine. Change any variable, and the throughput drops. Worse, the tooling is unstable. dFlash, vLLM, MLX—they all have quirks. One update breaks the drafter. Another OOMs the GPU. The developer experience is reminiscent of early Linux GPU drivers: powerful, but requiring a PhD in configuration.
Local AI isn’t about speed; it’s about stability.
I’m not here to trash the effort. The progress is incredible. Apple Silicon’s unified memory is a genuine advantage—you can run models that would require a $5,000 GPU cluster on a single Mac. But the current obsession with raw token speed is dangerous. It misleads developers into thinking production-ready local AI is here. It’s not. Not yet.
What should you actually care about? Memory bandwidth, first. Then tooling stability. Then context window management. Token speed is a vanity metric. The real metric is: how many minutes of uninterrupted conversation can you get before a crash?
If you’re building on local AI today, hedge your bets. Use the cloud for complex tasks, and keep local for simple, private ones. The hybrid approach is the only reliable path. And when the frameworks mature—and they will—the world will change. But for now, respect the wall.
The future of local AI is not about speed; it’s about trust. And right now, the software doesn’t trust itself.
FAQ
Q: Isn't 52 tok/s enough for most use cases?
A: For simple chat or single-turn tasks, yes. But complex tasks like chain-of-thought reasoning, multi-turn conversations, or drafting require stable memory and context management. Real-world effective throughput is often half that, and crashes reset the context.
Q: What should I focus on if I want to run local AI on Apple Silicon?
A: Focus on memory bandwidth and software stability. Use the MLX framework with nvfp4 quantization, but test with your actual workload—not synthetic benchmarks. Expect to iterate on tooling and accept that not all models will run reliably. For production, consider a hybrid local-cloud approach.
Q: Is cloud AI always better than local AI?
A: No. Local AI offers privacy, zero latency, and no API costs for simple tasks. For complex models, cloud is still more reliable. The future is hybrid: local for quick, private tasks; cloud for heavy lifting. The current hype around local AI ignores the software maturity gap.