Video-IFBench Tests Whether Video MLLMs Actually Follow Instructions

Introduction

A video multimodal large language model may recognize objects, describe events, or answer questions correctly, yet still fail to follow the user’s actual instructions. Real interactions often combine several requirements: use evidence from both frames and sound, answer multiple questions, select an output path based on what happens in the video, and obey a requested format. Conventional video benchmarks usually emphasize whether the final answer is correct. They rarely measure whether every part of the instruction was followed.

Video-IFBench is designed to isolate this missing capability. Instead of treating video understanding as a single question-answering problem, it evaluates whether a model can combine content interpretation with constraint-aware execution.

Key points

• Four instruction templates: The benchmark includes single-task, multi-task, selection, and nested instructions. The latter structures require models to compose operations or enter the right branch after inspecting the video.
• Audio-visual grounding: Constraints can depend on visual events as well as audio content, making the tasks closer to real multimedia interactions.
• A detailed taxonomy: The benchmark covers 32 task types and 39 manually designed constraint categories. These include semantic requirements and surface-level output requirements such as format.
• Semi-automatic construction: The authors combine MLLMs, programmatic processing, and human verification to reduce annotation effort while producing roughly 1.5K samples.
• Broad model evaluation: More than 20 recent MLLMs are evaluated, with attention to instruction adherence rather than answer accuracy alone.

What the results show

The reported findings indicate that video instruction following remains challenging for current systems. Performance becomes less reliable when a prompt contains many simultaneous constraints. Semantic constraints are also difficult because a model must first interpret an event correctly and then translate that interpretation into the exact action or answer requested by the user.

Conditional instructions expose another weakness. When a prompt asks the model to choose a branch or path according to video content, the model may identify some relevant evidence but fail to use it to control the next step. This distinction matters: an error may not mean that the model completely misunderstood the video. It may instead reflect incomplete planning, branch selection, or execution.

Why it matters

Video-IFBench makes a useful separation between understanding and reliable interaction. For video assistants, multimedia search, and multimodal agents, omitting one constraint, ignoring audio evidence, or violating the requested format can make an otherwise plausible response unusable. Its taxonomy can also help researchers diagnose whether failures arise from perception, semantic reasoning, or instruction execution.

The benchmark should nevertheless be interpreted with care. The reported dataset contains about 1.5K samples, and the available material presents overall observations rather than detailed scores or a model ranking. Video-IFBench is therefore best viewed as a diagnostic evaluation of instruction following, not as a complete measure of video intelligence. The work also highlights why video question-answering accuracy alone may not predict real-world reliability.

Source: Hugging Face Daily Papers