GAMUT reframes factual evaluation around completeness

Introduction

Most factuality evaluation for long-form generation has been built around precision: does the model’s answer contain false claims? The common decompose-search-verify pipeline is useful for catching inaccuracies, but it says much less about a different failure mode: did the model omit information that a complete answer should contain? GAMUT, introduced by researchers from Meta AI, focuses on this missing half of factuality.

Key points

• Completeness, not just correctness: A response can be factually accurate and still fail because it leaves out essential details. GAMUT evaluates whether the answer covers the required information, not only whether stated claims are true.
• Two-level meta-rubrics: The framework first represents the expected answer as a structured meta-rubric, capturing the organization and importance of content. That structure is then mechanically compiled into a flat set of binary, machine-gradable checklist items.
• Better fit for open-ended tasks: Many long-form answers are not simple lists of independent facts. They may involve open-ended coverage, ordered procedures, or relationships among facts. The two-level design preserves that structure before translating it into judgeable checks.
• Benchmark scale and grounding: GAMUT contains 1,813 questions grounded in real wearable imagery across 10 diverse domains. Each question is paired with an evidence-backed rubric verified by expert human annotators.
• Modality-agnostic design: Although the benchmark is instantiated around multimodal factuality, the framework is not tied to images. The authors also release a text-only variant.
• Model evaluation: The study evaluates 14 frontier and open-weight models. According to the provided material, the benchmark is genuinely challenging, discriminative and robust to judge choice, with the best reported score being 58.7% from Gemini 3.1 Pro.

Why it matters

GAMUT changes the question from “Did the model hallucinate?” to “Did the model give a sufficiently complete answer?” That distinction is important for real-world use. In education, medical explanation, operational guidance or visual understanding, an answer that is correct but incomplete can still mislead users or leave them unable to act.

The two-level rubric design is also a practical compromise. Expert annotators define the high-level structure of a complete answer, while the system turns that structure into binary checks that an LLM judge can score more reliably at scale. This is more expressive than a flat checklist and more scalable than fully manual grading.

The approach still depends on the quality of rubrics and expert verification, so cost, coverage and potential rubric bias remain open issues. Even so, GAMUT highlights an important direction for evaluation: factuality is not only about avoiding false statements, but also about not omitting the facts that matter.

Source: Hugging Face Daily Papers