UK AISI and EvalEval Push for More Reproducible AI Evaluations

Introduction

Benchmark scores increasingly influence research conclusions, product claims, and policy discussions. Yet the same benchmark name can hide substantial differences in prompts, inference budgets, feedback mechanisms, task handling, and statistical aggregation. As a result, two reported scores may not be directly comparable, even when they appear to measure the same capability.

The UK AI Security Institute (AISI) and the EvalEval Coalition are addressing this problem through a new phase of their collaboration. AISI is making selected evaluation methods and findings available through EvalEval’s Evaluation Cards, including verified results, configuration details, and surrounding context. The release accompanies AISI’s paper, How Inference Compute Shapes Frontier LLM Evaluation, which studies how inference-time compute and evaluation protocols affect frontier-model performance.

Key points

• Five benchmarks are included in the main experiment. The release covers HealthBench, FrontierMath, Humanity’s Last Exam, SWE-Bench Pro, and Terminal-Bench 2.0.
• Six frontier models are represented. The listed results cover Claude Opus 4, Claude Opus 4.5, Claude Opus 4.6, GPT-5, GPT-5.2, and GPT-5.4. Two related cyber evaluations, Cyber CTFs and The Last Ones, use a different and partially overlapping model set.
• The record goes beyond a final score. Evaluation Cards are designed to combine benchmark metadata, evaluation-run data, model metadata, and setup information in a common structure.
• Transcript-level detail can support diagnosis. For tasks involving repeated attempts or feedback, the underlying interaction record can help researchers understand how a result was reached, not just whether a task was eventually solved.
• The data follows the Every Eval Ever schema. EvalEval’s shared schema is intended to reduce information loss when evaluation results move between papers, leaderboards, repositories, and platform pages.

Why protocols matter

The material uses Humanity’s Last Exam to illustrate how performance can change with both inference compute and evaluation design. In one protocol, models received correctness feedback from an oracle after each attempt. As the token allowance increased, models continued to solve additional tasks. The point is not that one protocol is universally preferable, but that the protocol itself becomes part of the result.

A score therefore cannot fully describe an evaluation. Researchers may also need to know the task version, inference limits, retry policy, feedback availability, and method used to aggregate successful attempts. Without that information, a difference between two reports can be mistakenly attributed to model capability when it actually comes from experimental setup.

Implications

The collaboration’s main contribution is infrastructure rather than another leaderboard. By publishing verified results alongside the information needed to interpret them, AISI creates reference points that other researchers can inspect and compare in context. This is particularly useful where existing reports do not provide enough detail to reproduce or diagnose their findings.

For model developers, Every Eval Ever offers a structured way to report verified evaluation results. Benchmark developers can use the schema to share benchmark and run data. Evaluation, governance, and policy researchers can explore Evaluation Cards by model or benchmark and examine broader patterns in reporting quality.

The initiative will not make every evaluation immediately reproducible, and it does not make scores from different setups automatically comparable. Its more practical effect is to make those differences visible. If more evaluation organizations adopt shared reporting conventions, meta-research can move beyond collecting headline numbers toward understanding the conditions that produced them.

Source: Hugging Face Blog