WorldMemBench Tests Whether Video World Models Remember Their Worlds

As video generation systems move toward longer and more interactive experiences, a convincing frame is no longer sufficient. A useful world model must remember what happened before and preserve the consequences of events that occurred outside the current view. If a character leaves the camera, does the model retain the same identity and appearance? If an object is moved, does its position remain correct when the scene is revisited? If an event changes the environment, does that change persist?

WorldMemBench is designed to measure these capabilities systematically. Rather than treating memory as a single visual-consistency score, it divides the problem into static memory and dynamic memory. The benchmark uses controlled camera departures and multi-revisit trajectories to examine what happens after the model temporarily loses direct visual access to part of the world.

The benchmark covers:

• Human memory: retention of identity and appearance.
• Object memory: consistency of visual attributes, placement, and geometry.
• Environment memory: preservation of scene content and spatial structure.
• Event memory: progression of events and persistence of resulting states.
• Diagnostic signals: video quality and camera-following accuracy, which help separate memory failures from general generation or control failures.

WorldMemBench contains 260 test cases and evaluates nine representative models. Its central observation is that strong static retention does not imply reliable dynamic evolution. A model may recreate a person or scene that looks similar to the initial reference while failing to preserve the state changes caused by an earlier event. The benchmark also shows why a single revisit can be misleading: models that appear acceptable on the first return may degrade after repeated visits.

This distinction matters for interactive video, games, virtual environments, and any application that expects a world to accumulate history. Users do not merely want a collection of visually similar clips. They expect the same world to respond to previous actions and to remain temporally coherent as the viewpoint changes. By combining appearance metrics, geometry metrics, and event-specific checks, the benchmark offers a more useful diagnosis than image similarity alone.

The authors also report that the automatic evaluations align with human judgments, supporting the practical value of the proposed measurements. The benchmark does not itself provide a solution for building persistent memory, and it should not be read as a complete measure of every aspect of world modeling. Its contribution is to establish a shared testbed that exposes where memory breaks down and encourages progress from short-term visual continuity toward durable, interactive world maintenance.

Source: Hugging Face Daily Papers