You’ve spent years optimizing your stateless proxies, stripping away layers, chasing microseconds. You’ve convinced yourself that adding a globally distributed database like Spanner is a death sentence for latency. But there’s a team at Modal that just proxied inference requests in 6 milliseconds using Pingora, Envoy, and Spanner — and they didn’t abandon state. They weaponized it.
Let that sink in. A stack that includes three heavyweight tools — a service mesh proxy, a reverse proxy framework, and a globally consistent SQL database — produced a number that most engineers would call impossible. The conventional wisdom says you drop complexity to gain speed. The truth is more humbling: complexity isn’t the problem. Indecision about state is.
The tension here is delicious. Spanner is designed for strong consistency across continents. Envoy is a battle-hardened data plane. Pingora lets you tweak every byte. Together they should, by all logic, add 50ms. So how do they get 6ms?
Most engineers assume the bottleneck is in the data plane — the actual forwarding of packets. So they strip it bare, write custom C++, avoid databases like the plague. But Modal figured out that the real bottleneck is in the control plane: the logic that decides where to send each request. When you’re proxying inference requests, every millisecond counts, but the routing decision itself is what kills you. If your load balancer has to query a remote cache or stale config, you’ve already lost.
Their solution is beautifully counterintuitive: synchronize routing state globally using Spanner, but do it aggressively in the control plane, so the data plane never has to think. The data plane (Pingora + Envoy) becomes a dumb, fast highway. The control plane (Spanner) does all the heavy lifting — routing tables, model placements, capacity tracking — updated in near real-time, then pushed as a pre-computed decision tree to every proxy node.
The result: the proxy nodes never have to ask for permission. They already know, with global consistency, exactly where to send each inference request. The latency penalty of Spanner is paid ahead of time, in the control plane, amortized across millions of requests. It’s not that Spanner is fast for decision-making; it’s that the decisions are made before the request arrives.
Here’s the golden quote that every systems architect should screenshot: “Your stateless proxy is a lie. State is the only path to speed — if you know where to put it.” The industry has been obsessed with statelessness because it’s easier to reason about. But when you need 6ms, stateless means every request pays for routing discovery. Stateful, done right, means the routing cost is prepaid.
I saw this firsthand in Modal’s architecture: they stripped the data plane to bare metal efficiency (Rust, custom TCP stacks), but they fed it from a globally synchronized control plane that never compromises on consistency. The twist? Most teams would have built the same stack and gotten 50ms. The difference wasn’t the tools — it was the realization that latency isn’t killed by complexity. It’s killed by indecision.
What does this mean for you? If you’re building any latency-sensitive distributed system — inference proxies, edge compute, CDN routing — stop obsessing over which proxy framework is faster. Start obsessing over how your control plane synchronizes state. The data plane is a solved problem. The control plane is where milliseconds go to die.
Modal’s 6ms result isn’t a hack. It’s a blueprint. They took the most “heavy” tools and made them fly by changing their relationship: the control plane does the thinking, the data plane does the sprinting. That’s the engineering thrill of hunting milliseconds — not with faster lumber, but with smarter sawing.
FAQ
Q: Doesn't adding Spanner introduce unavoidable latency from database writes?
A: Not if you separate control plane writes from data plane reads. Modal pre-computes routing decisions from Spanner's globally consistent state, then pushes them to proxy nodes as an efficient lookup table. The proxy never queries Spanner on the critical path.
Q: So I need Spanner to replicate this? What if I use a different strongly consistent database?
A: Spanner is one option. The principle is what matters: use any globally consistent store (CockroachDB, YugabyteDB) that can feed a centralized control plane. The latency gain comes from the architecture, not the vendor.
Q: But isn't stateless load balancing simpler and good enough for most use cases?
A: Good enough for 95% of cases. But when you need sub-10ms at scale, stateless means every request suffers a routing decision. Modal shows that a stateful control plane is actually simpler to optimize because you control the entire decision path—no surprises, no stale caches.