World in World Gives Frozen Video World Models More Flexible Control

Why this matters

A video world model should do more than predict the next frame. In an interactive setting, it should let a user explore a recorded event, change the camera, continue the scene over time, and return to an earlier location without losing visual identity. Autoregressive video models can produce long rollouts, but flexible control remains difficult. A new viewpoint must stay synchronized with the original event, visible content must appear in the requested location, newly exposed regions must be completed plausibly, and previously generated appearances should be recovered on a revisit.

The paper World in World addresses this problem with a training-free interface that operates at inference time. The underlying causal video model remains frozen; the main contribution is a way to organize and route control evidence through its existing attention mechanism.

How the interface works

• A shared evidence format: Source-video observations, target-view scene projections, geometry renderings for newly exposed subject regions, and retrieved generated states outside the rolling cache are converted into clean visual states labeled by camera and time.
• Native self-attention as the interface: Instead of adding a separate task-specific module for every control mode, the evidence is made available to the frozen model through its native self-attention.
• Correspondence routing: A router combines persistent point identities with geometry to establish token correspondences. Queries with reliable support are directed toward matching source-video tokens, helping preserve content as the viewpoint changes.
• Evidence-specific guidance: Evidence-wise attention CFG, or EWA, independently regulates the extra contribution of each auxiliary channel. Its guidance is derived from attention responses in the same denoising forward pass, rather than treating all conditions as one undifferentiated signal.

What it enables

The design targets several related forms of exploration. Camera-controlled rerendering can use observations and projections to place known content in a requested view. Geometry cues can guide completion where a subject becomes newly visible. Retrieved generated states can extend useful context beyond the model’s rolling cache, which is important when the explorer returns to an earlier part of a long trajectory. The same interface is also presented for human-motion transfer, suggesting that the evidence-routing mechanism is not limited to a single camera-control task.

The paper evaluates camera-controlled video rerendering under varied viewpoint changes, considering perceptual quality, temporal consistency, and camera-following accuracy. The supplied material does not include numerical results, so the contribution is best described as a general inference-time control framework rather than evidence that it universally replaces specialized training.

Broader significance

World in World reframes controllability as an evidence-organization problem. Instead of modifying a capable world model for every new task, a system can potentially combine visual observations, geometry, temporal availability, and memory through one shared interface. This could make interactive video simulation and explorable scene reconstruction easier to extend while preserving the capabilities of an existing backbone.

There are also clear dependencies. Performance still relies on the quality of the source video, geometric cues, point correspondences, retrieved states, and the frozen model’s own ability to synthesize unseen content. The supplied abstract does not establish how the approach behaves under extreme viewpoint changes or very long error accumulation. Those cases remain important directions for further evaluation.

Source: Hugging Face Daily Papers