The Billion-Dollar Bet That Might Be Wrong: Why AI’s Sacred Scaling Law Is Cracking

You’ve probably felt it. That nagging anxiety in the pit of your stomach every time you hear about another $100 billion compute cluster being built. The fear that you’re betting on the wrong horse. That your competitors are spending smarter, not just bigger. That the entire industry is marching in lockstep towards a cliff, and nobody is asking the obvious question: What if the map we’re all using is wrong?

For years, the AI industry has worshiped at the altar of a single, beautiful idea: Scaling Laws. The belief that if you just add more parameters, more data, more compute, the model’s performance will predictably—almost mathematically—improve. It turned multi-billion-dollar decisions into a simple formula. But what if the formula itself is built on a foundation of sand? What if the ‘law’ is less a law of nature and more a convenient bookkeeping error?

This is the bombshell that former OpenAI Safety VP 翁荔 (Lilian Weng) has just dropped on the industry. In two dense, devastating blog posts, she doesn’t just question the prevailing wisdom—she takes a pickaxe to its foundations. And what she uncovers isn’t just a flaw in the math. It’s an entirely new paradigm for where AI’s next leap forward will come from.

Let’s start with the sacred text. In 2020, OpenAI’s Jared Kaplan team published a landmark paper. They showed that if you plot a model’s parameters, training data, and compute on a log-log chart, the test loss falls in a near-perfect straight line. Bigger model = better results. It was a mathematical crystal ball. It told you exactly how much compute you needed to hit a target. This discovery directly guided the training of GPT-3: 175 billion parameters, 300 billion tokens. It was the scientific justification for the ‘bigger is better’ era.

Then came DeepMind’s Chinchilla team in 2022. Using the same theoretical framework, they ran a different set of experiments and came to the exact opposite conclusion. Their ‘optimal’ recipe said you should scale model and data proportionally. One framework. Two experiments. Two diametrically opposed ‘optimal’ strategies. If Kaplan was right, the entire industry had been training ‘data-starved’ models for two years. If Chinchilla was right, GPT-3 was a ‘malnourished giant.’

Weng does something most analysts missed. She doesn’t just accept the contradiction. She digs into the ‘bookkeeping.’ And she finds the rot.

The scaling curve is not a law of nature. It’s a bookkeeping artifact.

The first crack is in how you count parameters. Kaplan’s team excluded the ’embedding layer’ in their calculations. In small models, that’s a huge chunk of the network. Chinchilla included it. A technical accounting choice, right? Wrong. It completely distorted the shape of the fitted curve. When you re-calibrate using Kaplan’s method, the magic exponent predicting your model’s growth changes as the model gets bigger. The ‘law’ is not a straight line; it’s a curve that depends on how you hold the ruler.

Then Weng goes after Chinchilla’s own data. The Epoch AI team, trying to replicate their results, found two bugs. One was in how they calculated the ‘Huber loss’—they used the average instead of the sum, causing the optimizer to stop too early. The second is a killer: key parameters in the paper were only reported to two decimal places. Weng calculates that to achieve that precision, you’d need to run approximately 600,000 experiments. Chinchilla ran fewer than 500. The gap between ‘two decimal places’ and ‘600,000 experiments’ is the gap between a scientific law and a guess. That curve, used to plan multi-billion dollar training runs, is sitting on a foundation that is significantly softer than anyone wants to admit.

This isn’t just academic nitpicking. Weng built an interactive simulator to show how Scaling Laws fail. The results are terrifying. Tiny noise in the loss value—one-thousandth of a unit—can shift the entire curve. Rounding off a few decimal places changes the ‘optimal’ strategy from ‘scale the model’ to ‘scale the data.’ The errors are invisible in small-scale experiments, but they are exponentially amplified when you extrapolate to a trillion-parameter model. It’s like using a ruler calibrated at room temperature to measure a skyscraper that’s expanding in the heat. The ruler is fine. The context is not.

And then there’s the elephant in the room: the assumption that training data is infinite and unique. In 2020, that was plausible. In 2026, it’s a fantasy. The internet has been scraped dry. As data repeats, models begin to ‘remember’ instead of ‘learn.’ The ‘double descent’ phenomenon shows that test loss can actually increase as the model sees more repeated data. The assumption that ‘more data = better’ has a hidden expiry date. The industry is pivoting to synthetic data, reinforcement learning, and test-time compute precisely because the ‘infinite data’ premise of Scaling Laws is collapsing.

So if the old path is crumbling, where is the new one? Weng’s answer is the second part of her analysis, and it’s where the real excitement lies.

She calls it Harness Engineering.

You can’t just translate ‘Harness’ into Chinese. It’s not a framework, a toolchain, or middleware. Weng defines it as ‘the execution system built around the base model.’ It’s the system that orchestrates execution, decides how the model thinks and plans, calls tools and actions, manages context, stores artifacts, and evaluates results. In simple terms: The model thinks. The Harness acts. And then it feeds the results back to the model so it can think better next time.

Look at the success of coding agents like Claude Code, Codex, and Cursor. Their competitive edge isn’t their model weights. It’s their Harness design. When you give Codex a task, it doesn’t just output code. It enters a loop: observe the codebase, plan, read files, edit code, run tests, check for errors. If a test fails, it goes back to planning. That loop isn’t in the model’s weights. It’s the Harness. The model provides the raw intelligence. The Harness provides the structure for that intelligence to land in the real world and iterate.

Weng compares the Harness to an operating system. An OS encapsulates complex hardware logic and exposes a clean API to the application. The Harness does the same for the model—it encapsulates tool calls, state management, and error recovery, giving the model a clean interface to interact with the world. The future, she argues, will see these interfaces standardize, just like OS APIs did.

But the real magic of the Harness is the three feedback loops that enable true self-improvement: Feedback, Memory, and Strategy Update. Feedback is the starting point. The Agent must know if it did well. Weng insists that feedback must be verifiable—not just a model’s own subjective judgment. Did the code pass the test? Did the web page render? Did the tool call return the expected result? Memory is the storage. But not all memories are good. A good memory system filters, compresses, and structures experience. It doesn’t save everything; it saves the pattern. Strategy Update is the goal. The Agent doesn’t just remember; it changes its behavior. When these three loops connect, the agent forms a true improvement cycle.

Context Engineering is another critical piece. As agentic tasks get longer, simply appending everything to the context window is a recipe for disaster. Weng introduces ACE (Agentic Context Engineering) and its successor, MCE (Meta Context Engineering). MCE separates the ‘mechanism’ (how to manage context) from the ‘content’ (what’s in the context). It allows the system to evolve its own heuristics for managing memory, rather than relying on hand-coded rules. The system learns how to learn.

This is the true pivot. The industry’s narrative is shifting from ‘scale the model’ to ‘scale the system.’ The future of AI competition is not just a model competition. It’s a system competition. Model companies need to understand Scaling Laws to know where raw power delivers. Application companies need to understand the Harness to turn that potential into usable productivity. Most companies will need to bet on both lines simultaneously.

Weng’s two-part analysis is a wake-up call. It’s a warning that the map we’ve been using is flawed. But it’s also a map to the new frontier. The future of AI doesn’t belong to the company that trains the biggest model. It belongs to the company that builds the best Harness. The model is the engine. The Harness is the car. And for the first time, we’re realizing that a better car can beat a better engine, every single time.