NVIDIA Cosmos 3 Adds Four Open Models for Interactive World Simulation

A useful world model must do more than produce realistic-looking video. It should respond to actions, preserve relevant scene structure, and predict what an agent may observe next. NVIDIA’s latest Cosmos 3 release adds four open-weight Cosmos 3 Nano checkpoints aimed at that problem, with applications spanning autonomous driving and interactive robotics.

Key points

• Transfer-Auto generates controllable driving data. The model can produce synchronized video for up to 11 camera views, including fisheye cameras, together with jointly generated RGB and LiDAR data. World Scenario Map controls describe scene structure and traffic motion, while text prompts can vary appearance, weather, lighting, and time of day. It can also be post-trained for a new sensor rig or driving domain.
• Sim-Auto targets streaming closed-loop simulation. This causally distilled version of Transfer-Auto generates successive short chunks of synchronized camera and LiDAR observations from driving controls, scene controls, and generated history. A full closed loop still requires a simulation runtime and a WSM renderer to update controls from agent inputs.
• Sim-Transfer converts simulator state into visual observations. The physics simulator remains responsible for motion, contact, and physical state. The model uses spatial controls such as depth, segmentation, edges, or blur, along with text, to render RGB observations that a policy can consume.
• Sim-Bimanual predicts the consequences of robot actions. It supports YAM and AgiBot embodiments and can predict subsequent observations from an initial image and bimanual action sequences. The released interface includes conventions for coordinate frames, rotations, gripper calibration, validity masks, and timing.

Architecture and limitations

The four checkpoints are post-trained examples of Cosmos 3 Nano and contain 16B trainable parameters. They use OpenMDW1.1, so deployment must follow the terms of that license. Cosmos 3 connects text-processing and visual-generation networks through a Mixture-of-Transformers design. During streaming generation, the system predicts short video segments from controls and available history, using temporal-causal attention to block access to future observations. Compressed video latents and a key/value cache support continued generation.

Why it matters

The release is notable because it exposes several action-to-observation paths rather than treating video generation as a one-off rendering task. Driving policies can receive synchronized visual and 3D sensing data; robot policies can test the visual consequences of manipulation; and researchers can divide responsibilities between a physics simulator and a generative model.

Open weights and post-training recipes may make it easier to adapt the system to new sensors, robots, and domains. Still, the practical value will depend on evaluation: users must measure physical consistency, error accumulation over long rollouts, and whether generated observations are reliable enough for policy training or assessment.

Hugging Face Blog