You’ve seen the pitch. A thousand volunteers, each with a gaming GPU, pooling their power to train the next frontier AI. No more corporate gatekeepers. No more billion-dollar data centers. Just pure, beautiful democracy, crunching numbers in the dark.
It sounds like the future we deserve. It’s also a fantasy.
Here’s the hard truth: training an LLM is not mining Bitcoin. The physics of distributed optimization are fundamentally hostile to the kind of crowdsourcing that made crypto mining a reality.
I watched a friend try to build a distributed training pool. He had 50 RTX 4090s donated by enthusiasts across four continents. The first epoch took three weeks. The model collapsed into noise. The problem wasn’t the GPUs. It was the time they spent talking to each other.
In crypto mining, every hash is independent. Your GPU doesn’t care what my GPU is doing. They’re all screaming into the void, and the void just needs to hear one correct answer. That’s embarrassingly parallel—asynchronous, zero dependencies.
LLM training is the opposite. Every gradient update depends on every other gradient update. A single step requires synchronizing billions of parameters across every node in the network. If one node is 100 milliseconds late, the entire model waits. If one connection drops, the weights desync.
The real bottleneck isn’t the compute. It’s the high-speed interconnects. Nvidia’s NVLink and InfiniBand aren’t optional—they’re the invisible oligopoly nobody talks about.
You can buy a GPU on eBay. You can’t buy a 800 Gbps, sub-microsecond latency link between two continents. The internet today is built for YouTube, not for synchronous gradient descent. Even with perfect fiber, the speed of light imposes a 130-millisecond round trip between New York and Tokyo. That’s not a delay—it’s a dead stop for training.
And the populist dream of wresting AI’s future from Big Tech? It collides with cold, unforgiving physics. Decentralizing GPUs without decentralizing high-speed interconnects doesn’t create a democratic pool. It creates a fragmented, slow puddle.
Yes, there are projects—Petals, Hivemind, federated learning. They work for small models, fine-tuning, inference. But frontier training? The 100-billion-parameter beast? The math doesn’t bend. Communication overhead scales quadratically with model size. At some point, you’re not training—you’re simulating a data center that doesn’t exist yet.
This is why open-source AI still relies on centralized corporate compute. Meta can train Llama because they own the fabric. Mistral can train because they rent it. You can’t crowdsource a 100 Gbps, all-to-all, synchronized network from a thousand bedroom miners.
The irony is delicious. The same people who cheered for decentralized crypto are now bumping into the coldest, most centralized force in the universe: the speed of light and the physics of coherence. You can’t democratize synchronization. You can only rent it from the people who already own the switches.
So what does this mean for the future? It means the frontier will remain centralized until someone invents a new kind of distributed optimization—one that tolerates latency, handles partial synchrony, and doesn’t need every parameter to agree in real time. That’s not a software problem. It’s a fundamental research problem.
Until then, the dream of a people’s AI is a beautiful lie. And the hardest truth? The lie is comforting. The physics are not.
FAQ
Q: What stops people from just using the internet to train LLMs in a distributed way?
A: Latency and bandwidth. The internet is designed for asynchronous communication (like watching videos or sending emails). LLM training requires synchronous, all-to-all communication with sub-millisecond latency. The speed of light and network congestion make this impossible across wide geographic distances.
Q: So is there no hope for decentralized AI at all?
A: Not for training frontier models. But decentralized inference (running already-trained models) and fine-tuning smaller models are feasible. Projects like Petals show that. The bottleneck is only for the massive synchronous training runs needed for GPT-4 scale models.
Q: Could a new algorithm solve the synchronization problem?
A: Possibly, but it's a fundamental research challenge. Asynchronous gradient descent exists but degrades quality for large models. New approaches like local SGD or federated learning work for small models but haven't scaled to 100B+ parameters. Until someone cracks that, physics wins.