Do Video LLMs Really See? VDG Exposes Accuracy Without Grounding

Lead

Video large language models are often judged by benchmark accuracy: if a model answers questions about a video correctly, it is assumed to have understood the visual content. The paper “Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks” challenges that assumption. Auditing 20 models from 2B to 78B parameters across 10 architecture families, the study shows that being accurate on a benchmark and being visually grounded are not always the same thing.

Key points

• A new audit metric, VDG: The Visual Dependency Gap measures the difference in per-question correctness between the original-video condition and a black-screen condition. If a model remains correct after the video is removed, the question may not be testing visual understanding.
• Accuracy can separate from grounding: On MVBench, paired McNemar tests show significant differences among models on original videos, but not on black screens. This suggests that some answers may come from language priors, dataset patterns, or question biases rather than visual evidence.
• Task types behave differently: Attribute Perception is strongly visual, while Temporal Reasoning approaches a language-only baseline. In other words, some supposed video reasoning tasks may not force models to use temporal visual information.
• Frame diversity matters more than order: The authors introduce a diagnostic ladder: black screen, single frame, shuffled frames, and original video. Across 16 open-weight models, most of the visual benefit comes from seeing diverse frames, while preserving temporal order adds near-zero accuracy.
• Sparse sampling is not the main explanation: An ablation from 0.5 to 24 FPS rules out the simple claim that models fail to use temporal order only because they see too few frames.
• Compression can hide instability: H.264 experiments show that aggregate accuracy may look stable while individual questions flip in both directions, from correct to wrong and from wrong to correct.

Why it matters

The paper is not saying that video LLMs have made no progress. Instead, it highlights a measurement problem: a single benchmark accuracy score does not tell us whether a model actually used the video. A system may perform well on a leaderboard while still answering many questions correctly under black-screen conditions.

VDG offers a practical way to audit this issue. Remove the visual signal, compare per-question outcomes, and measure how much performance truly depends on video. A large gap indicates stronger visual dependency; a small gap raises questions about the benchmark design, answer distributions, or hidden linguistic shortcuts.

For model developers, this distinction matters because it separates “using visual evidence” from “exploiting benchmark regularities.” For benchmark designers, it suggests that future video evaluations should include tasks that cannot be solved by a single frame, shuffled frames, or language priors alone.

The diagnostic also extends to four API-accessed models, whose VDG values range from 0.025 to 0.315. That means the issue is not limited to open-weight systems. As video multimodal models move into content analysis, education, surveillance, and agentic workflows, evaluating whether they are truly visually grounded becomes essential for judging reliability.

Source: arXiv