You know the type. They call themselves rationalists. They have a whole community built around the idea that they think harder, deeper, and more rigorously than everyone else. They write long, carefully structured arguments. They use Bayesian reasoning in casual conversation. They are, by their own account, the most clear-headed people alive.
So you’d expect their foundational texts to be airtight. You’d expect data. Mechanisms. Evidence.
You’d be wrong.
The loudest voices warning us about AI intelligence have never actually demonstrated how it works. They’ve just asserted it — loudly, confidently, and with enough jargon to make you feel stupid for asking questions.
Eliezer Yudkowsky and Nate Soares — two of the most influential figures in AI alignment — produced a book that’s treated as scripture in rationalist circles. It’s cited, referenced, and relied upon by people who shape how billions of dollars flow into AI safety research. And here’s what’s inside: claims about how computers will achieve intelligence, presented as settled fact, with no empirical evidence backing them up. No datasets. No experiments. No mechanisms. Just reasoning that goes in a circle and calls itself a straight line.
Let me be clear about what’s happening here. This isn’t a minor academic quibble about footnotes. This is a community that built its entire identity on epistemic rigor — the idea that you follow the evidence wherever it leads, that you update your beliefs based on data, that you never confuse confidence with correctness. And their flagship publication violates every single one of those principles.
As one commenter put it bluntly: “Rationalists don’t need pesky things like data or evidence to assert their desired conclusions as fact. Don’t you know, you can assert your vibes as science, now.”
Ouch. But also — accurate.
When the people telling you the world is ending can’t show you the math, maybe the problem isn’t that you don’t understand the threat. Maybe the problem is that there isn’t any math to show.
Think about what’s actually at stake here. The AI alignment community doesn’t just write blog posts. They influence policy. They shape funding decisions. They advise governments and corporations on how to approach artificial intelligence. When Yudkowsky says AGI is an existential threat, people with real power listen. Resources get redirected. Research agendas get rewritten.
And all of it traces back to arguments that wouldn’t pass peer review in an undergraduate seminar.
Here’s the twist nobody wants to talk about. Most critics of the alignment community focus on whether their conclusions are right or wrong. Are they too alarmist? Too dismissive? Too extreme? But that’s the wrong argument entirely. The real problem is upstream of the conclusions. It’s the methodology. The rationalists have spent two decades telling everyone else to check their assumptions, question their priors, and demand evidence. And then they wrote a book that does none of those things.
The most dangerous thing about the AI alignment movement isn’t what they believe. It’s that they’ve convinced the world their beliefs are science when they’re actually theology.
I’m not saying AI safety doesn’t matter. I’m not saying we shouldn’t think carefully about artificial intelligence. What I’m saying is something more uncomfortable: the community that has positioned itself as the guardian of rational thinking about AI has a gaping hole where its evidence should be. And nobody — not the funders, not the policymakers, not the journalists — seems to have noticed.
If you follow AI safety debates, this matters to you directly. The dominant narrative about AI risk is shaped by people whose foundational text can’t pass its own community’s standards. That should change how you read everything they produce. Not because they’re necessarily wrong about everything — but because they’ve shown you their work, and the work is empty.
Next time someone tells you the rationalists have figured out AI, ask them one question: where’s the data? Watch what happens to their face. That expression — that flicker of uncertainty, that scramble for a citation that doesn’t exist — that’s what intellectual honesty looks like when it hits a wall it didn’t know was there.
You don’t get to claim the mantle of science and then skip the part where you show your work. That’s not rigor. That’s a costume.
FAQ
Q: Isn't it unfair to demand empirical evidence for something that hasn't happened yet?
A: No. If you can't provide evidence for your claims, you don't get to present them as settled fact. Speculation is fine — but label it speculation. The problem isn't the lack of data; it's the gap between the certainty of the claims and the absence of the evidence.
Q: What does this mean for AI safety funding and policy?
A: It means billions of dollars and major policy decisions are being shaped by arguments that haven't met basic scientific standards. If the foundational text lacks evidence, every downstream recommendation built on it deserves more scrutiny than it's currently getting.
Q: So you're saying the rationalists are just wrong about everything?
A: Not necessarily. They might be right about some conclusions. But being right for bad reasons is still a problem — especially when you've built your identity on caring about reasons. A broken clock is right twice a day, but you don't build a research agenda around it.