You built a bot. It trades on Kalshi. It pulls in endless streams of data, weighs probabilities, and executes trades with machine-like precision. You check the profit and loss sheet. It’s green. You feel like a genius.
But what if I told you a literal coin flip might be outsmarting your sophisticated algorithm?
Profitability is a liar. It tells you what worked, not what is true.
I recently scored a Kalshi prediction bot using the Brier score—a metric that measures the accuracy of probabilistic predictions. The result was agonizing. The bot wasn’t just lacking an edge. It was worse than a constant 50% guess. Its apparent advantage was completely illusory, and its probability estimates were actively miscalibrated. It was confidently wrong.
You’re probably thinking: ‘But it’s making money!’ Yes, it might be. But P&L can be fooled by position sizing and market luck. You can make a fortune on a single lucky trade while your underlying model is absolute garbage. The market doesn’t care if you’re right for the right reasons or right by accident.
A coin doesn’t get overconfident. It doesn’t fall in love with a narrative. It just flips.
When your bot processes thousands of data points to arrive at a 62% probability, it feels productive. It feels like an edge. But if the actual outcome happens 45% of the time, your sophisticated model is worse than ignoring all information and just guessing. The true benchmark isn’t other traders—it’s the naive forecast.
This is the uncomfortable reality of prediction markets. We build complex systems to parse news, sentiment, and trends. We want to believe our hard work translates to superior forecasting. But sophistication can actually reduce accuracy.
Complexity doesn’t just fail to add value; it actively subtracts it by dressing up noise as signal.
If you’re building or evaluating trading bots, forecasting systems, or any predictive model, you need to stop relying on P&L alone. Run a calibration check. Plot your predicted probabilities against actual outcomes. Use the Brier score. A calibration chart by probability bucket will show you exactly where your model is bluffing.
It hurts to realize your ‘smart’ system is outperformed by a coin flip. But catching that truth before it costs you real money? That’s the only real edge you have right now.
FAQ
Q: Isn't making money the whole point of a trading bot?
A: Until your luck runs out. P&L reflects past luck and position sizing, not predictive accuracy. A broken clock is right twice a day; a broken bot can hit a lucky streak right before blowing up an account.
Q: What exactly is a Brier score?
A: It's a metric that measures how close your predicted probabilities are to actual outcomes. A lower score means better accuracy. If your bot predicts a 60% chance of an event, and it happens 60% of the time, your score is good. If it happens 40% of the time, your model is worse than guessing.
Q: Does this mean all algorithmic trading is a scam?
A: No, but it means most 'edges' are imaginary. If your model can't beat a naive 50% baseline in a Brier score, you don't have alpha. You have a random number generator with a UI.