DeltaStreamer Gives Video Agents a Memory and a Reason to Look Back

Introduction

Streaming video creates a timing problem for multimodal systems. A video keeps arriving before anyone knows what a future question will ask about. The important clue may be a brief transition, an object that appears for only a few frames, or an action whose meaning depends on what happened earlier. If the system compresses the stream into broad textual summaries, that evidence may disappear before the question is asked.

DeltaStreamer is a framework built around a different assumption: watching a video and answering questions about it should be separate stages. During observation, the agent maintains a structured record of what changes. When a query arrives, it investigates the relevant part of the past and returns to the original visual evidence when its memory is not enough.

How the framework works

• It tracks changes, not only snapshots. DeltaStreamer compares consecutively sampled frames and records both the current scene and the visual differences between moments. This is intended to preserve event progression rather than reducing a sequence to a static description.
• It connects memory to source evidence. Observations are organized as structured textual memory with pointers to the corresponding video frames. The memory can therefore serve as an index for later visual verification instead of acting as the only source of truth.
• It plans around the query. After a question arrives, a query-driven planner identifies the evidence needed to answer it. The agent then retrieves relevant memory entries and determines whether additional inspection is necessary.
• It uses targeted replay to recover details. When a description omits a useful visual detail, the system revisits selected historical frames. This gives the agent a way to correct or enrich its memory after the initial observation phase.

Results and broader significance

The paper evaluates DeltaStreamer on OVO-Bench and StreamingBench, reporting overall scores of 66.07% and 59.53%, respectively, and describes improvements over representative streaming video understanding methods. The more important architectural idea is the combination of passive monitoring and active investigation. The agent does not need to preserve every visual detail in its textual memory at all times; it can retain an organized trail and spend additional computation only when a later question requires it.

That design is relevant to long-form video, livestream companions, surveillance-style monitoring, and interactive media assistants. The project page also provides open-source code and a locally deployable demo. A follow-up application, AI-Tongdan, applies the same general direction to livestream viewing by combining video, audio, and chat context.

The available material does not specify detailed latency, storage, replay-cost, or ablation results, so the benchmark scores alone should not be read as a complete account of real-time deployment trade-offs. Still, DeltaStreamer offers a useful way to frame the next stage of streaming video agents. The central question is no longer only whether a model can understand a frame. It is whether the system can decide what to remember, when to revisit the past, and how to ground an answer in evidence that may have been overlooked the first time.

Source: Hugging Face Daily Papers