You’ve probably noticed the victory laps. Every time a new frontier model drops, vLLM is there within hours, boasting support and shattering throughput records. It’s the undisputed king of AI inference. But if you’re actually building systems with it, you already know the dirty secret: we love vLLM, but damn if it isn’t frustratingly buggy.
The release of v0.28.0 is a perfect microcosm of the vLLM paradox. The community cheers the accelerating release cycle, but that same velocity is forcing compatibility patches, broken outputs, and missing features. It’s eroding the exact trust that made it the default in the first place.
Look past the benchmark charts and read the actual comments from people running this in production. One user recently tried running DeepSeek-V4-Flash on a B300. On v0.26, it was totally broken, requiring three out-of-tree patches just to function. They updated to v0.27—no patches necessary now, but the output was broken, randomly spitting out garbage.
Speed is a feature, but stability is the foundation. You can’t build a skyscraper on quicksand, no matter how fast you pour the concrete.
Then there are the silent omissions. Users hoping for a robust fix to the reasoning_content mess got nothing but a doc change. Loadtests are casually crashing. Sampler support? Still way behind. Where is support for top-n-sigma? For DRY? For XTC? C’mon guys.
And perhaps the most painful comment of all: “I just wish they’d support Pascal.” Nvidia might have given up on older architectures, but that doesn’t mean the open-source community has to. When vLLM drops legacy GPU support, it isn’t just abandoning old hardware; it’s abandoning the builders who can’t afford a fleet of H100s.
Most observers benchmark vLLM on raw throughput and new-hardware support, but the actual competitive battle is happening in the long tail: legacy GPU support, sampler diversity, and reliability at scale. This is exactly where llama.cpp’s slower, compatibility-first approach is quietly winning loyal users.
Innovation isn’t just about racing to the next horizon; it’s about making sure the bridges don’t burn behind you.
llama.cpp doesn’t have the flashy new-hardware headlines. It doesn’t break your outputs every minor version. It just works. It supports Pascal. It supports the weird samplers. It doesn’t require you to maintain a fork of patches just to get through a Tuesday afternoon.
The frustration you feel towards vLLM isn’t hate; it’s a sign of investment. We complain loudly because we rely on it. We want it to be better. But love doesn’t fix production outages. The community is tired of being beta testers for a project that claims to be production-ready.
vLLM is currently winning the sprint, but it’s bleeding out on the track. If they don’t slow down to fix the structural debt, the default engine of today will be the abandoned experiment of tomorrow.
The market doesn’t remember who shipped the fastest; it remembers who didn’t break when everyone else did.
FAQ
Q: If vLLM is so buggy, why is it still the industry default?
A: Because raw throughput on frontier hardware is currently the loudest metric in AI. People will endure immense technical debt if it means saving a few milliseconds on inference. But that tolerance is finite.
Q: What does this mean for my inference stack?
A: If you're deploying at scale or relying on legacy hardware and diverse samplers, you need a fallback. The hidden operational cost of maintaining out-of-tree patches for vLLM might outweigh the performance gains of switching to a slower, stable alternative.
Q: Is llama.cpp actually better than vLLM?
A: For the long tail of production use cases that require reliability, legacy support, and sampler diversity over raw speed—yes. vLLM wins the benchmark charts; llama.cpp wins the actual deployments.