Code2Video-Compass Tests What Code-Driven Video Models Can Really Do

Introduction

Video generation is moving beyond the simple pattern of entering a prompt and receiving a clip. In code-based generation, a multimodal large language model must interpret visual and temporal requirements, translate them into executable programs, and use those programs to produce a complete video. This creates opportunities for fine-grained control, targeted editing, and component reuse, but it also introduces failure modes that ordinary video-quality benchmarks may miss. Code2Video-Compass is designed to evaluate this broader workflow.

What the benchmark covers

• Three input settings. The benchmark includes text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V), allowing models to be compared across different forms of conditioning.
• A broad task portfolio. It contains 2,104 tasks across 10 categories and 24 subtypes. Task difficulty is described through scene construction, dynamic animation, and state dependency.
• Textual and visual references. Tasks are curated from written instructions and visual references, with corresponding T2V, I2V, and V2V tasks derived from complete professional and user-created videos.
• Process and outcome evaluation. The framework combines 12 objective metrics with five subjective metrics. It examines operational performance, video quality, text-video alignment, reference fidelity, instruction following, and overall quality judged by an MLLM.

What the evaluation shows

The study evaluates 15 recent proprietary and open-weight MLLMs from nine model families. Its findings point to significant performance gaps across input types and levels of task complexity. A model may produce an attractive-looking clip without consistently carrying out the requested code or operation. Conversely, reliable execution does not automatically guarantee strong visual quality or faithful adherence to the reference.

This separation between visual quality and execution reliability is one of the benchmark’s most useful observations. A code-based video system must do more than depict the right objects. It also needs to manage object states, actions, and temporal relationships in a way that remains faithful to the instruction. Ranking systems only by the appearance of their final clips can therefore conceal whether the underlying program actually completed the requested task, and whether the result is reproducible rather than accidental.

Why it matters

Code2Video-Compass places code-based video generation in a more realistic creative setting instead of treating it as a collection of narrow, specialized problems. Its cross-modal and multi-dimensional design gives researchers a way to identify whether failures originate in requirement interpretation, code execution, motion modeling, or reference preservation. For developers building applications, it also offers a practical reminder: a visually impressive sample is not enough. Execution stability and instruction completion matter just as much.

As code becomes an intermediate control layer for video generation, progress will be measured not only by the beauty of generated clips, but also by how reliably models turn complex visual intent into executable, reusable, and editable programs.

Source: Hugging Face Daily Papers