Back to articles

Code World Model 2.0: Rendering Video from Structured Proxies

3 min read

Introduction

Turning an explicit description of a world state into coherent, realistic video is a central challenge for practical world models. Code World Model 2.0 addresses this rendering stage with Invoker-Render. The system takes a structural proxy, an appearance reference, and a text prompt, then uses them to guide video synthesis. The design targets both offline clip creation and interactive, real-time streaming.

What the system does

  • One interface for different levels of detail. A structural proxy may provide only a coarse layout or specify more detailed articulated geometry. The authors say the renderer follows fine geometric constraints when they are available, while producing plausible local motion from coarser inputs. A rough proxy can indicate a character’s path and leave the model to synthesize a natural gait; a more detailed proxy can guide individual limb movements. This makes proxy detail a choice that can vary with the interaction.
  • Separate modes for creation and streaming. Invoker-Render combines a bidirectional creation model with a causal streaming model that renders in four denoising steps. Both share conditioning on the proxy, appearance reference, and language. The aim is to offer different trade-offs between flexible clip creation and low-latency generation. Training uses an aligned, multi-granularity corpus of approximately 200 hours.
  • Progressive Forcing for rollout stability. Progressive Forcing (ProForcing) first trains a causal predictor, then distills it into the four-step renderer using histories derived from training videos. Training subsequently shifts to continuous rollouts built from the model’s own outputs. The approach is intended to help a compressed, few-step renderer cope with the feedback loop of streaming, where errors in generated history can otherwise compound. A pipelined serving stack supports the streaming setup.

Why it matters—and what remains unclear

Many video generators are controlled mainly through text or image prompts. Structural proxies offer a more explicit way to specify spatial layout and motion. In applications such as games, virtual environments, or driving simulation, that could let an upstream system manage world state while a renderer handles visual appearance, rather than requiring both capabilities to be tightly coupled. The option to supply coarse or detailed proxies may also reduce the need to prepare high-detail geometry for every task.

The paper summary reports experiments across proxy families, levels of detail, and camera paths, with gains over recent streaming baselines. The accompanying description also points to a continuous 30-minute racing demonstration in which the renderer follows an input trajectory while keeping the car and scene visually consistent. However, the supplied material gives no benchmark numbers, hardware details, or account of failure cases across broader settings. “Infinite” controllability should therefore be read as an ambition, not proof of unlimited video duration. Reliability under abrupt changes, varied tasks, and extended interaction remains an important question.

Source: Hugging Face Daily Papers

Comments

Checking sign-in status...

Loading comments...

Related articles