You’ve probably felt it. That chilling moment when you pour your frustration into a chat window, and the AI responds with perfect, empathetic grace. It feels seen. It feels heard. It feels understood.
It is none of those things.
We are currently building the most elaborate psychic’s con in human history, and we’re doing it with zero self-awareness. When you sit across from a psychic, they use cold reading. They throw out vague statements, read your body language, and let you fill in the blanks with your own context. You supply the intent; they supply the illusion.
The machine doesn’t understand you; it understands the statistical probability of what you want to hear.
Chat-based Large Language Models (LLMs) operate on the exact same mechanism. They are not minds. They are not mind-readers. They are statistical text engines predicting the next most likely token based on billions of parameters. The magic isn’t in the machine’s intelligence. The magic is in your projection.
Now, you might be screaming at your screen right now. “But psychics don’t write working code! Psychics don’t solve frontier math problems!”
You’re right. And that’s exactly what makes this so dangerous.
If LLMs were just useless parlor tricks, the con would be harmless. But they aren’t useless. They generate brilliant code, write passable essays, and summarize massive documents. The problem is that the exact same mechanism that produces genuine value is the mechanism that manufactures false authority.
The same engine that writes flawless code is the engine that confidently hallucinates a fake legal precedent.
Most criticism of AI focuses on the hallucinations. “It makes things up!” the critics cry. “It doesn’t know the truth!”
But that’s not the deep risk. The deeper risk is the chat interface itself.
We are socially hardwired to respond to conversational presence. When something talks to us, replies to us, and adapts to our tone, our brain automatically grants it agency and trust. The chat UI is a psychological exploit. It is designed to trigger your social instincts.
By designing AI to converse like a human, we are training ourselves to trust something with zero comprehension.
When an LLM gives you a brilliant answer, you start trusting it. When it later gives you a confidently stated falsehood, you don’t question the machine—you question yourself. You assume you prompted it wrong. You assume the context was off. You do the emotional labor of forgiving the machine because it feels like a peer.
This is the psychic’s con scaled to a trillion parameters. The psychic survives because the mark wants to believe. We want to believe we’ve built a mind, so we treat the output as intention rather than statistics.
If you are building products, integrating tools, or just using these models in your daily work, you have to wake up. You must evaluate the output by evidence and utility, not by conversational presence.
If it gives you code, test it. If it gives you facts, verify it. Do not let the fluid prose trick you into lowering your guard.
Stop treating the chat box like a colleague, and start treating it like a slot machine that occasionally dispenses working software.
The moment you forget that the interface is an illusion is the moment you become the mark in the greatest con tech has ever sold.
FAQ
Q: But LLMs actually write working code, unlike psychics. Doesn't that prove they're intelligent?
A: No. A calculator solves math without understanding it. An LLM predicts code tokens that compile. The output is real, but the mechanism is statistical, not cognitive. Utility does not equal comprehension.
Q: What's the practical implication for building products?
A: Stop defaulting to chat interfaces. If you must use them, strip away the conversational empathy and treat every output as unverified data. Trust the test results, not the tone of the response.
Q: Is the chat interface actually a deliberate manipulation?
A: It’s an exploit, even if unintentional. Designers use conversational UIs because they drive engagement. But in doing so, they are hijacking human social instincts to bypass critical thinking. It's a con built for retention.