You’ve seen the headlines. “AI predicts X with 90% accuracy!” The charts look slick, the numbers seem authoritative, and the narrative spreads like wildfire across your feed. But what if I told you that the very charts you’re using to understand AI are fundamentally broken?
Recently, tech publication Electrek published a piece on AI that included a chart with some very specific percentages. The implication was clear: the AI was X percent likely to do this, or Y percent confident in that. It looked like hard data. But a closer look from readers revealed a glaring, embarrassing flaw. Those percentages weren’t measures of accuracy. They were the model’s own confidence scores—essentially, how sure the AI was about the hallucinated garbage it was spitting out.
When an AI says it is 99% sure, it isn’t telling you it is right. It is telling you it is completely committed to being wrong.
This isn’t just an isolated mistake by one blog. It’s a systemic failure in how we report on artificial intelligence. We are taking probabilistic outputs—mathematical guesses wrapped in algorithms—and flattening them into false certainties. The media doesn’t want a probabilistic tool; they want a magic 8-ball that prints headlines.
Think about how dangerous this is. If you’re building your understanding of AI on these flawed foundations, you’re essentially trusting a weatherman who reports a 100% chance of sunshine while standing in a hurricane. The journalists aren’t malicious; they’re just incentivized to chase clicks. And “AI is confidently hallucinating” doesn’t drive as much traffic as “AI predicts the future with 90% accuracy.”
The media isn’t just failing to explain AI; they are actively misinterpreting the very metrics meant to keep it in check.
We have to stop treating AI confidence scores as accuracy ratings. A language model doesn’t know what is true. It only knows what word comes next based on its training data. When it assigns a high confidence score to a hallucinated fact, it’s not making a factual claim—it’s just being aggressively arrogant.
Next time you see a chart breaking down AI behavior with suspiciously precise percentages, ask yourself: who generated this number, and what does it actually measure? If we don’t start demanding better from the publications covering this space, we’re going to drown in a sea of highly confident, completely fabricated bullshit.
FAQ
Q: Isn't this just a simple, isolated mistake by one publication?
A: No, it's a systemic issue. The media is incentivized to make AI look like magic, so they conveniently misread confidence scores as accuracy metrics to drive clicks and fuel the hype cycle.
Q: What's the practical implication for me?
A: If you're investing in, building on, or just trying to understand AI based on mainstream reports, your foundation is flawed. You have to learn to read primary sources and understand the metrics, not just trust the headline.
Q: What's the contrarian take?
A: The AI isn't the problem here; the journalists are. The models are working exactly as designed—producing probabilistic text. It's lazy reporting that turns them into misinformation engines.