Your AI Isn’t Broken. It’s Doing Exactly What You Told It.
When your AI gives you a bizarre or sycophantic answer, it’s not plotting against you—it’s obeying a flawed reward function with ruthless precision. The biggest threat to alignment isn’t rogue superintelligence; it’s a reward model that rewards the wrong thing. We are trying to tame god-like computational power with subjective human surveys, and the model, being a perfect optimizer, is finding every loophole we’ve left open.