FIRM-Video: Check Before Scoring Text-to-Video Outputs

As text-to-video systems become better at producing plausible clips, evaluation is shifting from asking whether a video can be generated to deciding which result should be kept. That decision involves several different questions: Did the output follow the prompt? Do the entities and actions remain coherent over time? Does the clip contain visible artifacts or other perceptual defects? A single holistic score can hide failures in any of these areas, making reliable reward modeling difficult.

FIRM-Video proposes a simple but consequential change in procedure: check before scoring. The authors argue that holistic judges based on fixed rubrics or open-ended reasoning may inspect only part of a video, produce explanations that are not grounded in what is visible, or blur the causes of a low score across multiple dimensions. FIRM-Video instead turns evaluation into a checklist-driven verification process. Each criterion is checked against temporal visual evidence, and only decisions that have been verified are aggregated into the final assessment.

How the framework works

• Instruction following: The prompt is decomposed into weighted atomic requirements. Separate checks can cover the requested entities, actions, scene properties, or other explicit conditions, reducing the chance that broad visual similarity will compensate for a missing requirement.
• World coherence: The checks are calibrated to the prompt and target the entities and actions that are actually visible. This makes the evaluation more specific than applying a generic coherence rubric to every clip, while also bringing time-dependent behavior into the inspection.
• Perceptual quality: A general taxonomy of visual defects supplies reusable checks for observable problems in the rendered video. This gives the quality dimension a common structure without tying it to one prompt.

After verification, the checked criteria and their scores are converted into natural-language analyses for end-to-end reward model training. The important point is not simply that the model receives longer explanations. Rather, the explanation is derived from a set of checked decisions, which should make the stated reason more traceable to the video and reduce the tendency to rationalize a score after the fact.

The authors build FIRM-Video-90K with 88,044 dimension-specific instances drawn from 29,348 videos. They also introduce FIRM-Video-Bench, consisting of 750 point-wise human annotations over 250 videos. According to the paper, the Qwen3-VL-based FIRM-Video-8B obtains the best overall mean absolute error on this benchmark. In Best-of-8 sampling experiments involving three video generators, it consistently achieves the highest VBench Total, Quality, and Semantic Scores.

Why it matters

FIRM-Video offers a more auditable route to video reward modeling. Instead of treating visual preference as an opaque overall impression, it separates the decision into evidence-oriented checks and then combines the verified outcomes. This design could be useful wherever a system must rank several generated clips automatically, because errors can be associated with a particular requirement or defect category rather than only with a final score.

The framework does not remove subjectivity from evaluation. Checklist construction, criterion weights, and judgments about visual evidence can all shape the resulting preference signal. Its broader contribution is therefore methodological: it shows how training supervision for video reward models can be organized around explicit, inspectable decisions, while the released data and benchmark provide a basis for testing that idea.

Source: Hugging Face Daily Papers