Beyond Video Understanding: A Benchmark for Video-Centered Deep Research Agents

Introduction

Deep research agents have become increasingly capable of searching the open web, collecting sources, and producing long-form reports. Most existing evaluations, however, begin with a text question or a set of web documents. A video-centered research task is different: the agent must understand what happens in the source video, decide which moments deserve further investigation, retrieve supporting or contrasting evidence, and make the final report easy to verify against the original footage.

The paper Beyond Watching and Searching: Benchmarking Agents for Video-centered Deep Research Reports proposes a benchmark for exactly this setting. Its focus is not simply whether an agent can describe a video, but whether it can build a research workflow around one.

The task goes beyond video QA

The proposed task, called Video-Centered Deep Research Report Generation, starts with a video and a user request. An agent is expected to identify relevant research questions, inspect audiovisual content, search for external multimodal evidence, and produce an interactive report linked to useful video moments.

That distinction matters. A conventional video question-answering system may return a short answer about what appears on screen. A research agent must provide a more traceable result. It should help users locate a relevant segment, check whether a claim is supported by outside sources, reproduce a demonstrated procedure, or make a better-informed decision based on both the video and the broader evidence base.

How OmniDR-Bench evaluates agents

The authors introduce OmniDR-Bench with 50 Chinese and English tasks spanning 19 real-world subcategories. The benchmark contains 6,414 criteria. Instead of relying only on a single holistic judgment, the study uses Omni Atomic Rubric Trees, or Omni-ARTs, to break evaluation into fine-grained checks.

The rubric trees examine three broad dimensions:

• Audiovisual grounding: whether claims are tied to the correct people, actions, speech, events, and moments in the video;
• Multimodal evidence use: whether the agent retrieves relevant external material and connects web, image, or video evidence to the source video appropriately;
• Interactive presentation: whether the resulting report supports navigation, verification, and access to the relevant timestamps.

Each tree includes 23 shared directions as well as task-specific binary criteria. To make the rubrics more dependable, the authors describe an Audited Rubric Construction process involving independent evidence collection, automated checks, behavioral calibration, and repeated human review.

What the results show

Across 18 agent configurations, the strongest system reaches 63.3 out of 100. The score suggests that current models can already combine parts of video understanding, web retrieval, and report generation, but they do not yet perform the full workflow consistently.

The reported weaknesses are especially instructive. Agents need broader coverage across video categories, stronger causal and logical analysis across modalities, and more accurate citation support. Finding a related web page is not enough: the source must actually support the specific claim, and the report must make the relationship between the claim, the source, and the video moment clear.

Why this benchmark matters

OmniDR-Bench shifts the evaluation target from “can an agent describe a video?” to “can an agent conduct research anchored in a video?” That framing is relevant to tutorials, product demonstrations, interviews, educational media, scientific procedures, and other settings where users need both explanation and verification.

The broader lesson is that multimodal research is not solved by adding more input types alone. Reliable systems need temporal grounding, cross-modal reasoning, source checking, and an interface that preserves the path from conclusion back to evidence. Benchmarks such as this one can make those requirements visible and give future agent systems a more realistic target than generic video summarization.

Source: Hugging Face Daily Papers