WOVEN: Teaching Multimodal Models to Reason About Visual State Changes

Introduction

Multimodal large language models can recognize objects and describe video, but they remain unreliable when asked to reason about what changes in a scene. Typical failures include predicting an object's location after a movement, tracking the consequences of an action, maintaining physical consistency through an occlusion, or determining the temporal order of events. These abilities are often studied separately under spatial, embodied, physical, or temporal reasoning benchmarks.

WOVEN proposes that these weaknesses may share a common source: a deficit in visual transition reasoning. The capability means inferring a later visual state from the current scene and the action or event that occurs between them. The paper treats this as a reusable world-modeling primitive rather than as another isolated task, and asks whether it can be learned from different forms of supervision and reused across downstream problems.

What the study contributes

• A structured source of supervision. WOVEN contains 36,076 examples organized across 20 scene types, 5 action types, and 8 reasoning types. The examples are generated from diverse and realistic rollouts produced by video-pretrained generative models. The central target is not simply identifying what appears in a frame, but explaining how the scene evolves.
• A broad diagnosis of current models. The authors evaluate 38 frontier MLLMs, including GPT-5.4 and Qwen3-VL-235B-A22B. Even the strongest systems remain substantially below human performance on visual transition reasoning. The same pattern appears across model families, and increasing scale does not remove the deficit.
• Transfer beyond the training source. Models trained at multiple scales on WOVEN improve 22 of 26 external benchmarks, with gains of up to 27.3 percentage points. The paper reports that subsets of roughly 2,000 examples can contribute to these improvements collectively. On some tasks, WOVEN data can replace 30–50% of the task-specific training data while retaining comparable accuracy.
• Clues for curriculum design. The study reports that models acquire physical principles in an order resembling infant development, and that this order can serve as a curriculum. Geometric perturbations damage performance much more than appearance perturbations, suggesting that spatial structure matters more than superficial visual variation. The authors also report that a few hours of generated-video training can outperform large-scale real-video mid-training on the same backbone.

Why it matters

The main implication is a shift in how multimodal capabilities might be trained. If visual transition reasoning is genuinely shared by spatial, temporal, embodied, and physical tasks, researchers could first teach a model to track state changes and then reuse that representation for video question answering, action planning, robotics, and scene prediction. This could reduce the need to build entirely separate supervision pipelines for every benchmark.

The claims should nevertheless be interpreted within the evidence provided. The experiments demonstrate transfer on the reported external benchmarks, not a complete solution to world modeling. WOVEN also relies on generated video, so the gap between synthetic rollouts and the complexity of real environments remains important. The contribution is best understood as a concrete framework for testing whether transition-centered supervision can serve as a common foundation for multimodal reasoning.

Source: Hugging Face Daily Papers