Omni-InteractScope Finds Persistent Gaps in Proactive Multimodal Interaction

Introduction

A multimodal assistant that continuously watches video and listens to audio faces a harder problem than ordinary question answering. It must decide whether to intervene, when to intervene, what action to take, and whether further follow-up is needed. Most existing evaluations focus on isolated prompts or single-turn answers, leaving the timing and continuity of real-time assistance poorly measured.

The paper Omni-InteractScope addresses this gap with a benchmark designed around proactive and agentic interaction. It links the available modalities with event triggers, intervention types, and user needs, testing whether a model can turn evidence from an ongoing stream into a timely and sustained response.

Key findings

• A broader interaction scope. The benchmark includes 1,479 human-annotated and cross-reviewed interactions, with 1,790 temporal targets distributed across eight task groups. It covers more than explicit questions: notifications, warnings, state tracking, continuous responses, query-free assistance, and event-timed tool invocation are all included.
• Tool use is still poorly timed. Among the 12 evaluated models, none produced an on-time first response for more than 40% of tool-invocation targets. When both timing and tool correctness were required, the best score reached only 32.0%. The result separates two capabilities that are often conflated: being able to call a tool and knowing when the call is justified.
• Proactive reminders lag behind question answering. Every evaluated model achieved lower timely reference-target recall on query-free reminders than on explicit questions, using the respective target sets. Reminder recall never exceeded 23.0%. In practice, a model may detect relevant content without recognizing that the user should be alerted without asking first.
• End-to-end guidance remains rare. The benchmark contains 26 guidance sequences that require multiple related responses over time. Nine models completed none of them in full, while the other three completed only one to three sequences each. This suggests that maintaining state and following through on a task are still difficult even when individual responses appear plausible.
• Tool scores are not enough to establish grounding. Targeted controls found that tool-invocation performance alone does not demonstrate event grounding. Reminder coverage also depends on response history and frequency, making a single success-rate metric inadequate for evaluating streaming assistants.

Why it matters

The benchmark shifts the question from “Can the model answer correctly?” to “Can it act appropriately in a changing environment?” For wearables, in-car assistants, real-time companions, and embodied systems, an intervention that arrives too early may be distracting, while one that arrives too late may be useless. Missing an initial event can also break every later step in a guidance sequence.

The findings point toward a broader evaluation agenda. Future streaming assistants need reliable event localization, timing control, user-need inference, response-frequency management, and stateful follow-up. Evaluation should therefore measure complete interaction chains: whether the system responds at the right time, selects the right action, uses prior responses appropriately, and ultimately covers the user’s goal. Proactive intelligence is not simply better perception or a longer context window; it is the reliable connection between perception, decision-making, and continued execution.

Source: Hugging Face Daily Papers