How Should Long-Video Models Spend Visual Tokens?

The central limitation in long-video understanding is often not that a model cannot interpret a frame, but that it never gets to see the relevant frame in the first place. Sampling an hour of video at one frame per second produces 3,600 images. Since context length, visual-token budgets, and inference costs are limited, most systems must retain only a small subset. That makes allocation strategy a first-class modeling decision: should the system spend its budget on a few detailed images, or on more moments at lower spatial resolution?

This paper addresses the question with a controlled comparison. Instead of evaluating bundled systems that simultaneously change the frame scorer, prompt boundary, resolution policy, and answering model, the authors hold those factors fixed and vary selection, spatial compression, and reinvestment separately. The study covers six training-free selection rules, three long-video benchmarks, and two answering models. This design makes it easier to attribute an accuracy change to the decision actually being tested.

Key findings

• Frame selection is the largest lever. On the hour-long split of LongVideoBench, eight query-selected frames outperform sixteen uniformly spaced frames by 6.9 percentage points. The result challenges the assumption that simply increasing the number of sampled frames guarantees better video understanding. Relevance and evidence coverage matter more than raw count.
• A classic algorithm remains surprisingly strong. Orthogonal Matching Pursuit, an unmodified sparse-approximation method developed decades ago, matches or comes within one point of every purpose-built selector in the comparison across all three benchmarks. A sophisticated, task-specific selector is not automatically necessary.
• Spatial compression is nearly free at fixed timestamps. Halving the spatial-token budget of each frame costs no more than 0.44 points in the reported experiments. Under a fixed context budget, preserving a smaller visual representation for each frame can therefore be a reasonable tradeoff.
• Reinvestment is the real source of the gain. Compression by itself is not the main win. When the saved tokens are used to double the number of frames, while keeping the total visual budget comparable, accuracy improves by another two to three points. Compression works because it enables broader temporal coverage, not because unused savings have value on their own.

Why it matters

The paper suggests a practical hierarchy for long-video inference. First, select moments that are likely to answer the query. Next, reduce per-frame spatial detail only as much as needed. Finally, spend the resulting savings on additional timestamps that can reveal missing events, transitions, or evidence. In other words, the optimization target should be evidence coverage under a visual-token budget rather than maximum quality per image.

The study also highlights why implementation discipline matters in selector research. The authors report a bug in their own AKS baseline and observe a 0.07-to-3.74-point gap between two answering models. Such details can materially affect comparisons, making controlled ablations, implementation checks, and cross-model evaluation as important as proposing a new selection mechanism.

The findings should still be read within their experimental scope. They come from particular benchmarks, models, and budget settings, so they do not establish one universal compression ratio for every video task. They do, however, provide a clear design principle: when visual tokens are scarce, spend less on redundant spatial detail and more on carefully chosen temporal evidence.

Source: Hugging Face Daily Papers