Code-as-World: Turning Raw Video into Executable Physics Simulations with MuJoCo
The gap between visual perception and physical understanding has long been a bottleneck in AI. A model might predict what a scene looks like next, but without internalizing concepts like mass, contact, or gravity, it's merely generating plausible pixels—not understanding the world. MirroS's new paradigm, Code-as-World, challenges this status quo by representing physical scenes not as pixels or latent vectors, but as executable code.
Instead of attempting to extract meaning directly from video frames, Code-as-World converts them into a scene.json program that MuJoCo, a physics engine, can run. This executable representation is more than just a simulation; it's a verifiable, editable, and re-runnable model of reality. An agentic loop refines these programs against the source footage in up to five iterations, ensuring accuracy. The result is a dataset of worlds with exact physical labels—ground truth that raw video simply doesn't provide.
This supervised training approach has produced Code-as-World-VL-9B, a model achieving a 55.4 MRA (Mean Relative Accuracy) score on the QuantiPhy validation set. This performance edge, roughly 15 points above the strongest open-weight baselines and surpassing Gemini-3.1 Flash (54.8), signals a significant step forward for physical AI.
Is It Ready for Deployment?
For those eager to experiment, the answer is a qualified yes. MirroS has released the project under an Apache 2.0 license, providing both the GitHub repository and two fine-tuned checkpoints—Code-as-World-VL-4B and Code-as-World-VL-9B—built on Qwen3.5-4B and Qwen3.5-9B, respectively.
These models are ready for immediate use in research and internal prototyping. They're packaged as BF16 safetensors, deployable via vLLM behind an OpenAI-compatible /v1 endpoint. The setup supports processing 16 sampled frames per video, with a recommended --max-model-len of 4608 tokens. This accessibility makes it feasible for developers and researchers to validate the approach in real-world applications, moving from theoretical promise to practical exploration.
Why This Matters in 2026
As we move further into an era of embodied AI and robotics, the demand for scalable, accurate physics understanding is becoming critical. 2026 is seeing a convergence of large multimodal models and simulation, with the industry moving from passive perception to active, verifiable world models. Code-as-World arrives at this crossroads, offering a tangible bridge between high-level language instructions and low-level physics simulation. By treating physical scenes as code, it enables not just prediction, but intervention—allowing an AI to reason, edit, and re-simulate scenarios, a capability that is foundational for next-generation robotics and autonomous systems.
via MarkTechPost