StreamVibe: Video Agents That Track Evolving User Intent

Introduction

Many vision-language models still follow a passive interaction pattern: a user asks a clear question, the model analyzes the current frame, and then produces an answer. Real-world assistance is less explicit. Someone navigating a website may pause before the next click without asking for help. A useful assistant must infer the likely need from ongoing visual observations and user behavior, while also deciding whether intervention is appropriate at that moment. StreamVibe studies this form of proactive, streaming interaction.

A benchmark for intent-aware assistance

The work introduces StreamVibe-Bench as a testbed for understanding and interacting with user intent. It contains 4.1K evaluation samples and 85K training samples, covering three settings: explicit intents stated by the user, implicit intents inferred from behavior and visual context, and dynamic intents that evolve during an interaction.

This framing goes beyond conventional single-turn video question answering. A streaming assistant must track more than the answer to a question. It needs to maintain a working hypothesis about the user’s goal, update that hypothesis as the scene changes, and determine whether and when to speak. These timing and state-tracking requirements are central to practical proactive systems.

StreamVibe-VL and StreamVibe-Harness

Built on the benchmark, StreamVibe-VL is designed to infer implicit intent from continuous observations and adjust its interaction as that intent evolves. According to the paper, it retains strong general video-understanding capabilities while improving reported response-timing and answer-quality performance by 10.7% over existing baselines.

The authors also introduce StreamVibe-Harness, which represents user intents as persistent states with flexible lifecycles. Its hierarchical memory tracks the latest intent together with the environmental state. The goal is not to preserve every past observation equally, but to give the system a structured way to update its current understanding during sustained interaction. The paper reports a further 12.3% gain in proactive interaction performance, along with better response consistency and adaptation.

Key takeaways

• Evaluation expands from answering questions to tracking intent over time.
• Implicit intent requires evidence from user behavior, visual context, and scene changes.
• Dynamic intent requires continuous state updates rather than a fixed initial interpretation.
• Proactive assistance depends on deciding whether and when to respond, not only on answer accuracy.
• Hierarchical memory provides a mechanism for maintaining intent and environmental state across an ongoing interaction.

Why it matters

StreamVibe highlights a shift in how video agents should be evaluated. A system that acts only after an explicit request is closer to a passive question-answering tool. A real assistant must understand context without becoming intrusive, respond when assistance is useful, and revise its interpretation when the user’s goal changes.

The results also expose a gap in current proactive models: implicit and dynamic intents remain difficult, particularly when the model must coordinate intent understanding with response timing. The benchmark and model design offered by StreamVibe provide a concrete way to study that gap. They do not eliminate the risks of mistaken assistance, excessive interruption, or conflicting states, but they move video agents toward a more realistic objective: continuously understanding a user and acting at the right moment.

Source: Hugging Face Daily Papers