APM-Bench Rethinks Persistent Memory for Egocentric Video Assistants

Introduction

A smart-glasses assistant that only understands one uninterrupted recording is closer to a short-term observer than a personal assistant. Real usage is fragmented: a device may be turned off between interactions, and a user may return hours or days later with a question whose answer depends on something seen previously. APM-Bench is designed around this setting, focusing on how egocentric streaming video assistants store and reuse memory across separate sessions.

Beyond single-video understanding

Many existing streaming-video benchmarks study one continuous video or a short clip. They measure whether a model can recognize objects, understand actions, or answer questions while the relevant context remains in the current stream. That setup leaves out several practical questions. What should be saved when a session ends? When should an old memory be brought into a new interaction? And if the required evidence was never retained, can the assistant admit that it does not know instead of filling the gap with a plausible guess?

APM-Bench reformulates real-world interaction as multi-session life trajectories. It contains 549 sessions, 104 trajectories, and 2,719 candidates. Each session is a finely annotated video, while sessions within the same trajectory are connected by related activities. The benchmark includes both questions with objective answers and open-ended questions that require broader contextual reasoning. It also examines whether an assistant can use prior experience to offer proactive responses during a later interaction.

What the benchmark measures

• Cross-session recall: Can a model locate relevant visual evidence from an earlier session?
• Memory management: Can it store information and selectively retain what is likely to matter under finite capacity?
• Timing of memory injection: When should historical context enter the reasoning process without overwhelming the current interaction?
• Operational efficiency: What are the gains in answer quality, and what latency and storage costs do they introduce?
• Evidence awareness: Can the assistant recognize that necessary evidence is unavailable and respond accordingly?

The study evaluates general video models under different memory protocols and compares specialized streaming-memory systems. Its main message is a clear utility–latency–storage trade-off. Improving long-range recall often requires additional computation or storage, while aggressive compression and low overhead can reduce reliability.

Why it matters

APM-Bench is more than another video question-answering set. It treats useful memory as a pipeline involving writing, retention, retrieval, injection, and calibrated refusal. For a personal assistant, remembering everything is neither realistic nor necessarily desirable. The important capability is to preserve evidence that may matter later, retrieve it at the right moment, and remain honest when the evidence is missing.

This framing also points to a systems-level research agenda. Memory cannot simply mean placing a longer video history in the model’s context window; it must work together with real-time perception and interaction policies. As streaming visual assistants move toward wearable and long-term personal applications, cross-session evaluation such as APM-Bench can become a more meaningful measure of practical usefulness.

Source: Hugging Face Daily Papers