LatentStream Moves Streaming Video Memory Beyond Retrieval

Introduction

Understanding a continuously arriving video is a different problem from analyzing a single image or a short, pre-segmented clip. A multimodal large language model must reason only from what has already appeared, while retaining enough historical information to answer questions that arrive much later. A person, object, or action seen several minutes earlier may be essential to the final answer, yet the model cannot keep every frame in its active context.

A common solution is to compress past observations into an external memory bank. When a query arrives, the system retrieves relevant clips or visual summaries and adds them back as context. This reduces the immediate context burden, but it also leaves historical evidence outside the model’s working state. Reasoning therefore remains dependent on repeatedly searching and reading an external archive.

The proposed approach

The paper introduces LatentStream to shift streaming memory from “store-and-retrieve” toward “retrieve-and-internalize.” The goal is not simply to enlarge the memory bank, but to transform retrieved evidence into a compact latent working memory that can continue guiding reasoning.

• Hierarchical history organization. Query-agnostic Hierarchical Streaming Memory divides visual history into short-term, mid-term, and long-term levels under a fixed budget. A Jenks-guided adaptive consolidation strategy is used to organize information across different temporal scales rather than applying one uniform compression rule.
• Progressively wider access. Once a query is received, Hierarchical Latent Memory Evolution assigns groups of latent memory tokens progressively expanding receptive fields. Each group retrieves evidence from its relevant scope, internalizes it into a fixed-length latent state, and then gains access to a broader historical range.
• Confidence-guided refinement. Progressive Confidence-guided Latent Memory Optimization builds a hierarchical progression reward from group-wise predictive entropy. The latent memory is optimized jointly so that its evolution is guided not only by retrieved content, but also by the model’s changing confidence.

Why it matters

LatentStream treats memory as more than a passive archive. It makes memory an evolving working state that can participate directly in later inference. A fixed-length latent representation could reduce the repeated movement of visual context through a long-video pipeline and provide a common mechanism for online and offline understanding.

The available material does not include benchmark scores, dataset-level comparisons, or ablation details, so the exact source of the reported gains cannot be independently assessed here. The approach also raises important questions. Compression may discard details that become relevant to a later query; an early mistake may be written into the latent state; and different questions might encourage different forms of internalization. These issues will be important for evaluating the reliability of latent memory in real streaming systems.

Even with those open questions, the paper highlights a meaningful design transition: video systems may need to evolve from merely preserving searchable history to continuously building an internal memory that supports ongoing reasoning.

Source: Hugging Face Daily Papers