If Code Builds the World, Can Video Models Just Render It?

Introduction

Video generation models are usually asked to solve two problems at once: decide what exists in a scene and render it as a coherent sequence. Multimodal coding agents suggest a different architecture. Code can construct an executable 4D world with objects, motion, spatial relationships, and camera paths, while a video model acts as a visual renderer that turns this specification into a desired appearance.

WorldDraft asks whether current models can reliably perform that rendering role. Rather than judging only visual plausibility, the benchmark separates two requirements. World preservation means retaining the prescribed 4D kinematics and camera trajectory. Visual realization means consistently binding the target attributes described by text throughout the generated sequence.

A controlled benchmark

WorldDraft includes 432 strictly paired test cases across 48 executable 4D world families and six types of dynamic challenges. Its 3×3 factorial design independently varies two factors. The first is draft granularity, ranging from an abstract proxy to a previs-style draft. The second is appearance alignment, from neutral visual evidence to evidence that conflicts with the target description. The dynamics remain identical across the paired settings, making it easier to determine whether a model preserves the underlying world or simply copies the reference appearance.

The evaluations produce three notable findings:

• Preservation and realization are in tension. A model can maintain motion and camera behavior yet fail to apply the requested visual change, or follow the appearance instruction while disturbing the prescribed dynamics.
• More detailed scaffolding can backfire. Rich drafts anchor scene structure, but they also provide more textures and stylistic cues for the model to copy. As a result, structural reliability may rise while appearance fidelity falls.
• Conflicting cues expose source leakage. When draft appearance and textual targets disagree, models can carry properties from the source into the output. The reported leakage reaches as high as 64%, pointing to fragile cross-modal disentanglement.

A practical mitigation

The authors also introduce WorldDraft-Primer, a plug-and-play visual prompting strategy. It is designed to clarify which information in the draft should be preserved and which attributes should be replaced, without modifying or retraining the frozen video renderers. Across the reported evaluations, the approach improves both world preservation and visual realization.

Why it matters

The broader contribution of WorldDraft is an interface for combining programmatic simulation with generative video. Instead of forcing one model to invent physics, control a camera, and design appearance simultaneously, a future pipeline could let code provide an executable and testable world state while a video model supplies visual realization.

That division of labor is not automatically reliable. A detailed draft is useful only if the model can distinguish geometry and motion constraints from appearance cues that the text asks it to change. WorldDraft makes this distinction measurable and highlights source leakage as a central obstacle to controllable video rendering. Its results suggest that progress in this area will require not only better video quality, but also stronger cross-modal separation and evaluation of temporal consistency.

Source: Hugging Face Daily Papers