Everyone in AI is obsessed with the same thing: architecture. Transformer variants, mixture-of-experts, rotary embeddings, attention tricks. The papers, the tweets, the GitHub repos — they all live at the model layer. And that’s exactly why almost everyone is leaving 40-60% of their performance on the table.
The most powerful optimization in AI today isn’t a new architecture. It’s rewriting the math that runs on the metal.
I’m talking about GPU kernels — the low-level computational primitives that actually execute your model’s operations. Matrix multiplies, attention computations, normalization passes. Every single thing your model does, a kernel does it. And most practitioners have never touched one.
Here’s the uncomfortable truth: when you call torch.matmul or F.softmax, you’re accepting someone else’s optimization ceiling. You’re driving a car with the parking brake partially engaged and wondering why it feels sluggish.
The AI community has created a bizarre paradox. We have the most sophisticated model architectures in human history — and we’re running them through generic, one-size-fits-all kernels designed to work across every hardware configuration, every tensor shape, every use case. That generality is a tax. And it’s a tax you’re paying in latency, in GPU hours, in cold hard cash.
Convenience and performance aren’t on a spectrum. They’re on opposite ends of a tug-of-war, and convenience has been winning while your inference bill quietly doubles.
Consider what happens when someone actually writes a custom kernel. FlashAttention didn’t change the transformer architecture — it rewrote how attention is computed in memory. The result? 2-4x speedup on long sequences. Triton, OpenAI’s kernel-writing language, didn’t invent new math — it made it possible for humans to express math in a way GPUs actually like. The models didn’t change. The kernels did.
And here’s the twist nobody talks about: kernel optimization isn’t just about speed. It’s about access. A well-optimized kernel can mean the difference between running a frontier model on an A100 cluster versus running it on a single consumer GPU. That’s not a performance gain. That’s a democratization of compute.
Every time you can’t deploy a model because it’s ‘too expensive to run,’ the problem isn’t your model. It’s your kernels.
The developers who understand this have an unfair advantage. While everyone else is fighting over marginal architecture improvements — squeezing 3% from a new activation function — kernel engineers are doubling throughput. They’re not publishing papers. They’re shipping code that makes existing models run like they got a hardware upgrade for free.
If you build or deploy AI models, here’s the reframe: stop asking ‘what architecture should I use?’ and start asking ‘what kernels are actually running, and who wrote them?’ The answer might be the most expensive assumption in your entire stack.
The model layer is where ideas live. The kernel layer is where reality lives. And reality doesn’t care about your paper’s benchmarks.
FAQ
Q: Isn't writing custom kernels too hard and niche to be worth it?
A: That was true five years ago. Tools like Triton and CUDA Graphs have dramatically lowered the barrier. You don't need to be a GPU architect — you need to understand your model's bottleneck and write a kernel that removes it. The ROI is often 2-4x speedup for days of work, not months.
Q: What does this mean for teams deploying models in production?
A: Audit your kernel stack before you scale your GPU fleet. If you're running stock PyTorch ops on production workloads, you're likely overpaying for compute by 30-50%. Custom or community-optimized kernels (FlashAttention, FlashInfer, vLLM's PagedAttention) can cut inference costs dramatically without touching your model.
Q: Is architecture innovation actually overrated then?
A: Not overrated — over-indexed. Architecture determines what's possible. Kernels determine what's practical. The field has poured 95% of its energy into the former and 5% into the latter. That imbalance is the opportunity. The next wave of AI efficiency gains will come from the layer everyone's ignoring.