Back to articles

VETO Targets Textual Shortcuts in VideoLLM Fine-Tuning

3 min read

Introduction

Supervised fine-tuning (SFT) is a common way to improve a video large language model’s ability to answer questions about clips. But a higher score does not necessarily mean the model has become better at interpreting the video. It may also learn recurring textual patterns in its training examples and use them to predict an answer without relying much on the visual evidence. Such a shortcut can remain hidden when evaluation data resembles training data, then become costly when the distribution changes.

The paper introduces VETO, short for Video-Evidence Training via Orthogonal projection. Rather than trying to identify and remove every shortcut-bearing example, it intervenes in the fine-tuning update itself, with the stated goal of retaining SFT’s gains while reducing reliance on text-only cues.

How the approach works

  • The problem is not easy to define at the data level. According to the paper’s summary, textual shortcuts can arise across different training epochs, dataset sizes, and model backbones. Their presence may coexist with better accuracy on the training distribution, so that improvement alone does not show whether a model is grounding its answers in the video.
  • VETO constrains the update direction. The method calibrates a subspace associated with text-only input in the activations of the base model. It then projects the fine-tuning update onto the orthogonal complement of that subspace. In intuitive terms, it limits updates along directions associated with text-alone representations, without first having to specify exactly which textual regularity constitutes the shortcut.
  • The evaluation emphasizes shifted data. The authors report experiments across seven VideoLLMs: VETO retains SFT’s gains on the training distribution and suppresses the shortcut. On six benchmarks under distribution shift, it performs better than SFT and the models answer from the full input rather than relying on the shortcut. The supplied material does not give benchmark names, scores, or effect sizes, so the results cannot be quantified here.

Why it matters—and what remains open

The broader point is that task learning and shortcut learning can happen together. A model may appear capable because question wording or text co-occurrence patterns predict the label, even when its use of video evidence is limited. If so, familiar-distribution accuracy can overstate genuine video understanding. VETO offers a training-time intervention: instead of exhaustively describing the shortcut in the dataset, it constrains updates using a representation space associated with text-only input.

The evidence provided supports the method on the tested models and benchmarks; it does not establish that shortcuts are eliminated in every video task. The available summary also omits details such as how the subspace is calibrated, computational overhead, per-benchmark results, and ablations. Those details will matter when assessing reproducibility and deployment trade-offs. The authors have released code, making further evaluation possible.

Source: Hugging Face Daily Papers

Comments

Checking sign-in status...

Loading comments...

Related articles