You’ve probably been mesmerized by the latest AI video generators. You type a prompt, and a hyper-realistic clip of a dog playing poker or a car drifting through neon-lit streets appears. It feels like magic. It feels like the pinnacle of artificial intelligence.
But according to NVIDIA’s Max Li, who is helping build the company’s Cosmos foundation models, generating a pretty video is the biggest distraction in AI today.
Generating a beautiful video isn’t intelligence; it’s mimicry. True intelligence requires understanding the consequences of your actions.
We’ve been building AI backward. The industry standard right now is to train massive models to understand text and images, and then try to bolt on “action” at the end as a post-training optimization. It’s like teaching someone to read a map, but never letting them touch the steering wheel. The model can describe the world, but it doesn’t know how to move through it.
Max Li and the NVIDIA Cosmos team are flipping this script. They argue that “action”—the ability to interact with, manipulate, and change the physical environment—must be baked directly into the pre-training phase, right alongside video, text, and audio. If a model doesn’t understand how its actions alter the world, it’s not an agent. It’s just a glorified encyclopedia.
This is a radical shift in how we think about machine intelligence. For decades, robotics has been trapped in a cycle of highly specific, hard-to-reproduce hardware experiments. If a system can’t scale its data, it can’t scale its intelligence. But when you treat action as a core modality during pre-training, you allow the model to learn the fundamental physics of cause and effect at scale.
If a model only does what it’s explicitly programmed to do, it’s just a traditional script with a probability distribution slapped on top.
Real intelligence emerges when a system can break out of its pre-defined boundaries. It needs to understand a messy environment, adapt to changes, and complete tasks it was never explicitly coded to do. That’s the difference between an automated arm bolting on a car door, and a general robot agent figuring out how to clean your kitchen.
Of course, building this unified model is an engineering nightmare. You can’t just throw text, video, and action into a blender and hope for the best. Cosmos uses a Mixture of Transformers (MoT) architecture—text uses autoregressive methods, while video and action use diffusion. It’s a pragmatic compromise. Theoretical purity is great, but in the real world, engineering feasibility is what actually ships.
There’s a fascinating tension here: strong understanding clearly boosts generation capabilities. But right now, generation doesn’t necessarily boost understanding. The feedback loop is broken, and solving that is the next great frontier for AI researchers.
But make no mistake, NVIDIA isn’t building Cosmos to win a video generation demo contest. They are deliberately ignoring the flashy content creation hype.
Cosmos isn’t here to win a film festival. It’s here to become the CUDA of Physical AI.
It’s designed to be the open-source infrastructure layer that allows agents to finally step out of the computer and into our physical reality. Whether it’s robotaxis navigating unpredictable traffic, or humanoids adapting to human-centric environments, the underlying requirement is the same: a foundational model that understands the physical world.
The next decade of AI won’t be won by models that write the best poetry or generate the most viral memes. It will be won by models that can step into reality, understand the physics of a room, and take action to change it.
FAQ
Q: If action is just another modality, why can't we just fine-tune it later?
A: Because post-training action optimization doesn't give the model a fundamental understanding of cause and effect. If you bolt action on at the end, the model only knows how to react; it doesn't understand the physical consequences of its movements.
Q: What does this mean for the average developer?
A: It means the foundation for robotics and autonomous systems is shifting from hardware-specific code to scalable, pre-trained foundation models. Developers will soon build agents on top of physical AI infrastructure, much like they build apps on top of LLMs today.
Q: Is NVIDIA just trying to sell more GPUs by pushing Physical AI?
A: While it definitely drives GPU demand, the core technical argument holds weight. Video generation for content is a parlor trick; building models that understand physical reality is the actual hard problem that unlocks multi-trillion dollar industries like logistics and autonomous transport.