ATAS Gives Video Agents a Memory of Places and Time

Introduction

Video generation systems can already produce impressive individual shots, but assembling those shots into a coherent long-form story remains difficult. A room may change layout when the camera returns to it, while an action replayed from a different viewpoint may no longer correspond to the original moment. Passing prompts or a few recent frames between requests provides local context, but it does not create a shared spatial reference or a time-indexed record of what happened earlier.

Any Time, Any Space, or ATAS, proposed by a Zhejiang University team, treats this problem as a world-state management task. Rather than replacing the underlying video generator, the framework organizes screenplay parsing, evidence retrieval, clip reconstruction, quality review, and memory updates into a single agentic pipeline.

Key ideas

• Referenced shot requests: ATAS compiles a screenplay into requests that specify a location, camera information, and event time. This allows the system to distinguish returning to a place from replaying a particular event.
• Two complementary memories: Scene memory stores registered static geometry, such as the layout of an environment. Event memory stores dynamic observations indexed by scene, event, and time.
• Retrieval before generation: Before producing a shot, the agent searches the memories for the spatial and temporal evidence relevant to that request. Afterward, it reconstructs and registers the generated clip.
• Selective memory admission: A new observation is written to memory only when review and state checks accept it. Film delivery remains independent of whether the observation is admitted, reducing the risk that an unreliable clip will distort later shots.

Results and implications

The paper evaluates ATAS with four video generators. The authors report improvements on all listed cross-shot consistency measures and the highest aggregate benchmark score for each backbone. Additional experiments compare target-image spatial revisits and event recapture, alongside human judgments. In those evaluations, ATAS is reported to outperform the compared agentic and 3D-guided approaches, and to receive the highest human ratings across six criteria. These findings apply to the reported experimental setup; performance in practice will still depend on the capabilities of the base generator and the complexity of the scene.

The broader contribution is a clearer decomposition of continuity. Instead of treating consistency as an implicit property that the video model must remember, ATAS makes it an explicit workflow: store spatial structure, record dynamic events, retrieve the right evidence, and decide whether a new observation is trustworthy enough to become part of the world state. This could be useful for multi-shot production, interactive storytelling, and replayable video environments.

The approach also leaves open important questions. Memory reliability depends on reconstruction and state checking, and a two-memory design may face additional challenges when scenes contain changing geometry, occlusion, or complex long-term interactions. More evidence will be needed to understand how well the framework scales beyond the reported benchmarks.

Source: Hugging Face Daily Papers