OmniCompass Brings Spatiotemporal and Audio-Visual Rigor to Video Evaluation

Why another video benchmark?

Multimodal models can increasingly describe videos and answer questions about them. Yet a plausible answer does not necessarily mean that a model found the right evidence. It may fail to identify when an event occurs, where an object is located, or which audio cue supports an answer. OmniCompass is designed to evaluate these abilities together, turning video understanding into a more diagnosable set of grounding, perception, and reasoning problems.

A benchmark built around complementary evidence

The benchmark has two tracks. OmniCompass-VL targets models that primarily depend on visual and language inputs. OmniCompass-AV focuses on omni-modal models that must combine audio and visual evidence. Each track contains three tasks:

• Temporal grounding (TG): identify when a referenced target appears;
• Spatial grounding (SG): identify where the target is in the frame;
• Question answering (QA): determine what happens, covering perception and reasoning.

OmniCompass-VL contains 820 videos and 2,460 instances drawn from different domains and divided into short, medium, and long duration groups. Its narrative- and audio-rich AV subset contains 240 videos and 720 instances. Every instance is evaluated under paired text-only and text-plus-image specifications. The target remains the same, while the way it is specified changes, allowing the benchmark to test whether a model’s answer depends too heavily on the reference format.

The instances were manually annotated and subjected to multiple rounds of quality verification. The project reports approximately 1,800 hours of human annotation, indicating an emphasis on reliable labels as well as sample coverage.

What the evaluation exposes

The authors evaluate 50 multimodal large language models, measuring both task performance and consistency across target specifications. Precise spatiotemporal grounding and fine-grained perception remain major weaknesses. The gap between open-source and proprietary systems is particularly visible when tasks require audio and visual information to be combined.

Adding a visual reference is not universally beneficial. Its effect depends on the model and the task, and a high score under one specification does not guarantee correctness under another. Timestamps generally improve grounding and substantially stabilize evaluation, suggesting that explicit temporal cues can make both model behavior and measurement more reliable.

Why it matters

OmniCompass frames video understanding as an evidence chain: where a target is, when it appears, what happens, and whether the model can connect visual, linguistic, and audio signals consistently. For developers, this separation makes it easier to distinguish failures in perception, localization, and reasoning. For users, cross-specification consistency may be more informative than a single leaderboard score.

This perspective is relevant to video agents, multimedia retrieval, and interactive media systems. Such systems need more than fluent answers; they must point to the moments and regions that support those answers. OmniCompass therefore offers a useful reminder that one video-QA score should not be treated as a complete measure of video intelligence.

Source: Hugging Face Daily Papers