OmniScope: Decoupling Audio and Video Salience for Faster OmniLLM Inference

Introduction

Omnimodal large language models are increasingly expected to reason over text, video, and audio at the same time. The challenge is that audio-video inputs quickly produce a large number of tokens, which makes the prefill stage slower and increases GPU memory pressure. Token compression is therefore becoming a practical requirement for deploying these models, not just an academic optimization.

OmniScope focuses on a subtle but important flaw in many existing approaches: they often let one modality decide what should be kept in another. For example, visual cues may be used to guide audio pruning, or audio signals may influence which video tokens survive. The paper argues that this assumption can fail in real audio-video understanding.

Key ideas

• Audio and video salience can peak at different times: For the same user query, the most relevant sound and the most relevant visual evidence may appear in different temporal regions. Under aggressive compression, one-way cross-modal guidance can remove tokens that are essential for answering.
• Use the query as the common anchor: OmniScope keeps the query as a shared semantic reference, but it estimates relevance separately for audio and video. In other words, the modalities share the question, not the salience map.
• Allocate token budgets by modality: Instead of applying a single pruning rule over all tokens, the framework assigns modality-specific retention budgets.
• Visual compression with anchor-delta pruning: For visual tokens, OmniScope uses an anchor-delta strategy designed to preserve both global context and temporal changes.
• Audio compression through within-second merging: For audio, it merges tokens within each second to reduce redundancy while maintaining temporal continuity.

Results and impact

Across four audio-video benchmarks and two Qwen2.5-Omni model scales, OmniScope reports the best average accuracy across compression settings. At 25% overall token retention, it achieves up to 3.53x prefill speedup and more than 15% GPU memory reduction, with only a 0.35-point drop in average accuracy.

The broader lesson is straightforward: omnimodal inference should respect the fact that modalities are synchronized in time but not necessarily synchronized in relevance. A spoken clue may matter before or after the visual clue, and a compression algorithm that assumes otherwise can discard the wrong evidence.

Because OmniScope is training-free, its appeal is also practical. It suggests a lightweight path to faster inference without requiring model retraining. This could matter for long audio-video understanding, interactive assistants, and deployment scenarios where GPU memory is limited. The available summary does not reveal all implementation details or per-task behavior, so the full paper and code remain important for deeper evaluation. Still, the design principle is clear: share the query across modalities, but keep audio and video salience estimation decoupled.

Source: Hugging Face Daily Papers