What Does an Evaluation Result Actually License?

Introduction

A benchmark number is often treated as if it carries its own interpretation. Once a script produces a score, readers may immediately ask which model is better, whether an evaluator is reliable, or whether a historical comparison has been reproduced. Yet an evaluation artifact normally specifies only a forward computation: a task, a scorer, and a reported metric. It may not preserve the historical evidence or semantic context required to replay the claim attached to that metric.

“What Does an Evaluation License?” makes this gap explicit. Instead of asking only whether Inspect Evals can run, the paper asks what the available evidence actually permits an auditor to conclude. The analysis is pinned to a particular commit and treats claim replay as a separate layer of the evaluation process.

Key points

• An executable artifact is not the same as a licensed claim. Code can define how a value is computed without fixing which historical inputs or meanings should be used.
• The claim layer is formalized. The audit introduces a frozen substrate D, a grounded family F of admissible interpretations, and a claim query q. Their resulting identified set contains the outcomes still compatible with the evidence and semantics.
• Every eligible unit receives a terminal disposition. The census covers 124 mechanically eligible Inspect Evals units. In 110 cases, the process stopped before deterministic inference because required historical evidence or semantic grounding was unavailable.
• Resolution changes the answer. Among cases where execution closes, an exact value, a winning model, a complete ordering, and a pairwise relation are different claims. Stability at one resolution does not guarantee stability at another.
• Evaluation families matter. Results are separated for primary and review families rather than compressed into a single robust-versus-not-robust judgment.

Why it matters

This framing expands reproducibility from “the code still runs” to “the original conclusion is still supported.” In model leaderboards and automated judging, repositories are often preserved more reliably than historical inputs, version context, or the operational meaning of a label. Running the current artifact may therefore produce a valid new computation without reproducing the original historical assertion.

For evaluation designers, the practical lesson is to archive not only code and metrics but also pinned versions, historical evidence, scoring semantics, and the intended resolution of each claim. For evaluation users, a statement such as “Model A wins” should trigger a second question: does that mean an exact score, a complete ranking, or only a relation that survives across several admissible interpretations?

The paper advocates a fail-closed audit. Missing evidence or unresolved semantics should yield a typed stop, not an improvised guess. When plausible interpretations disagree, the audit should expose an instability witness; when they agree, it can report that agreement as stable substructure. The result may be less headline-friendly than a single leaderboard, but it is more precise about what an evaluation genuinely licenses.

Source: Hugging Face Daily Papers