If you’re building or training transformers, you’ve probably felt the weight of the feedforward layers. They’re expensive. They’re opaque. And for years, we’ve assumed they’re essential — the non-linear muscle that gives transformers their power. But what if we were wrong?
A new paper, Attention-Only Transformers, drops a bombshell: you can remove the feedforward layers entirely and still match — sometimes beat — the performance of standard transformers. The attention mechanism alone is enough. The feedforward layer is a crutch, not a necessity. This isn’t just a tweak; it’s a fundamental rethinking of what makes these models work.
Let that sink in. For nearly a decade, the AI community has built towers of feedforward parameters, convinced that depth and diversity were the keys to representation. Now, a stripped-down architecture — attention-only — challenges that orthodoxy. We’ve been adding complexity to solve problems that attention could already handle.
The paper’s insight is both elegant and disruptive. Attention mechanisms, it turns out, provide their own non-linearity through context-dependent weighting. The feedforward layers may have been redundant — a historical artifact from the early days of transformer design. This is the kind of result that makes you question everything you thought you knew about deep learning.
Of course, the skeptics will say: ‘But what about scaling? What about long-range dependencies?’ The authors address these head-on, showing that attention-only models can achieve comparable results on standard benchmarks. The real question isn’t whether it works — it’s whether we have the courage to abandon the architecture we’ve built our careers on.
For engineers and product builders, the implications are huge. Leaner models mean lower compute costs, faster inference, and more interpretability. The future of AI might not be bigger — it might be simpler. This paper forces us to reconsider the path of ever-increasing scale and complexity. Maybe the real breakthrough isn’t a new layer — it’s realizing we didn’t need the old one.
Read the paper yourself. Then ask: what else are we doing out of habit instead of necessity? The next wave of AI innovation won’t come from adding more — it will come from taking away.
FAQ
Q: Are attention-only transformers really as good as standard ones?
A: On standard benchmarks, yes. But real-world performance and scaling behaviors need further study. The paper shows strong results, but it's a proof of concept, not a production-ready replacement.
Q: What's the practical implication of this research?
A: This could reduce model size and compute cost by removing feedforward layers, making AI more accessible and faster to deploy. It opens the door to simpler, more interpretable architectures.
Q: What's the contrarian take on attention-only transformers?
A: The feedforward layers might be crucial for tasks not tested, like long-context reasoning or multi-modal learning. Attention-only could be a niche, not a universal solution. The paper's findings are provocative but not yet definitive.