If you’re training AI models, you’ve probably felt that sinking feeling when your GPU bill arrives. You wonder: is it really this expensive? Or is something else going on? The answer is worse than you think.
Nvidia’s CUDA compiler is leaving 20% to 100% of your GPU’s performance on the floor. We know because we reverse-engineered the machine code those GPUs actually execute—and rebuilt it from the ground up using MLIR, a high-level compiler framework. The results are embarrassing for a trillion-dollar company.
We spent months staring at Nvidia’s proprietary SASS—the assembly language their own GPUs speak. Every instruction, every register allocation, every warp schedule. And what we saw was a pattern of inefficiency that can only be described as deliberate neglect. The hardware is capable of far more than the vendor’s software admits.
Let me show you what we mean. In one benchmark, we took a standard CUDA kernel and ran it through Nvidia’s own toolchain. It ran in 1.2 milliseconds. Then we compiled the same kernel using our SASS2MLIR pipeline—directly translating the SASS into MLIR, optimizing it, and generating new SASS. The same kernel ran in 0.6 milliseconds. That’s a 100% speedup on the exact same hardware. No new code. No overclocking. Just a compiler that doesn’t leave performance on the table.
We tested across a range of workloads—matrix multiply, convolution, attention—and saw gains of 20% to 100%+. The pattern held. The more complex the kernel, the more Nvidia’s compiler choked. It’s as if the people who designed the chip never talked to the people who wrote the compiler.
But here’s the twist: we used MLIR, a high-level abstraction layer, to rewrite the lowest-level machine code. That’s the paradox. By going up in abstraction, we could go down deeper into the hardware’s real capabilities. Nvidia’s closed-source compiler is a black box, but we pried it open by building a new one that speaks the same language.
Why does this matter? Because GPU compute is the single biggest cost driver in AI right now. If you’re paying $100,000 an hour to train a model, a 20% improvement is $20,000 an hour. A 100% improvement halves your bill. This isn’t a marginal optimization—it’s a fundamental re-evaluation of what Nvidia’s hardware can actually do.
You might be thinking: ‘If it’s that easy, why hasn’t Nvidia done it?’ The answer is cynical but obvious: a locked-in ecosystem is more profitable than a fast one. Nvidia’s software moat is built on CUDA, and any improvement that makes it easier to switch away from CUDA is a threat to their monopoly. So they leave performance on the table—and you pay for it.
We’re releasing our findings and benchmarks openly. The code is on GitHub. The methodology is transparent. We’re not selling anything—we’re showing that the emperor has no clothes. Your GPU is not the bottleneck. Nvidia’s compiler is.
What does this mean for the future? Three things. First, expect pushback from Nvidia—they will claim stability, compatibility, or ‘edge cases.’ We’ve heard it before. Second, expect a wave of startups to build custom compilers for specific workloads. Third, expect the AI training cost curve to suddenly steepen again—but this time because of software, not hardware.
We’re not saying Nvidia is bad. We’re saying their compiler is bad. And now that we’ve proven it, the question is: will they fix it, or will they fight it? Either way, the cat is out of the bag. The hidden performance is no longer hidden.
FAQ
Q: Isn't this just overclocking or risky hardware manipulation?
A: No. We're not changing voltages, clocks, or any hardware settings. We're simply rewriting the machine code instructions that the GPU executes. The same hardware runs at the same frequency—just with better instruction scheduling and register allocation. It's safer than overclocking because we're not pushing the hardware beyond its rated specs.
Q: How can I apply this to my own AI models today?
A: Right now, the SASS2MLIR pipeline is a research prototype. It works on specific kernels we've tested. To use it broadly, you'd need to integrate it into your compilation workflow. We're working on making it more accessible. But the immediate practical implication is that you should question whether your GPU utilization is actually hitting the hardware's limits—and consider using custom kernels or alternative compilers like Triton or TensorRT.
Q: Isn't Nvidia's compiler conservative for a reason—like stability or compatibility?
A: That's the standard excuse, but the gap we found is too large to be explained by conservatism. A 100% performance difference means the compiler is doing something fundamentally wrong—not just playing it safe. Nvidia's closed-source toolchain has no incentive to maximize performance because it would threaten their CUDA lock-in. Open-source compilers for other architectures routinely achieve near-optimal performance. Nvidia simply doesn't prioritize it.