Stateful Harness Makes World Models Execute Rules Before Rendering Video

Introduction

Video world models are becoming increasingly capable of producing plausible continuations from a single image and a user action. They can let users move through a scene, inspect objects, or explore a generated environment. Yet visual plausibility is not the same as rule-consistent interaction. If an outcome depends on a hidden variable—such as a prior event or an unseen condition—two visually identical scenes may require different responses to the same action.

Stateful Harness, presented by a Tsinghua University team, addresses this gap by separating world evolution from video rendering. Instead of asking a video generator to remember and execute every rule, the framework gives state management to a dedicated harness and uses the video model primarily as a renderer.

Key points

• Rules update an explicit world state. The harness maintains an evolving representation of the environment and resolves the user’s selected action according to specified rules. This allows identical visible inputs to lead to different events when their hidden states differ.
• A frozen video model renders the event. After resolving an action, the harness compiles the resulting event into an instruction for a fixed video backend. A preview from an editable 3D scene supplies additional spatial and camera guidance.
• Generated clips feed the next turn. The output video is not treated as an isolated response. The system uses the generated clip to update the world state, creating a loop of action resolution, event rendering, and state transition.
• The benchmark tests counterfactual consistency. WorldState-Bench contains 50 counterfactual twin pairs. Each pair shares an initial frame and action but differs in hidden state and required ending, making it harder for a model to succeed through superficial visual matching.

Results and implications

Under rule-only prompting, the best result among 11 evaluated video backends produced the required endings for both members of a pair in 32% of cases. With MiniMax-H3, explicitly specifying the required endings increased that rate from 32% to 64%. The full Stateful Harness reached 92% with the same frozen model. The comparison indicates that the main bottleneck is not necessarily video quality alone. A generative model may be asked to perform state bookkeeping and rule execution that are better handled by an explicit control layer.

The design could be useful for interactive video, game-like environments, and embodied systems. Video generators can focus on appearance, motion, and camera behavior, while a stateful component handles decisions that can be inspected and evaluated. At the same time, the reported benchmark has 50 paired cases, and the framework depends on how world states and rules are represented. Larger and more open-ended tests will be needed to assess long-horizon interactions and the reliability of state updates.

Source: Hugging Face Daily Papers