BenchMIRT Asks What LLM Benchmarks Really Measure

Introduction

LLM evaluation is often reduced to a single number. A higher score is then treated as evidence that a model is better, safer, or more capable. Yet individual benchmark questions can require several things at once: reading comprehension, multi-step reasoning, factual knowledge, instruction following, and decisions about whether to refuse. Averaging those questions into one score can hide what is actually driving performance.

Ai2’s BenchMIRT is designed to audit benchmarks at the level of individual prompts. Its central question is simple: what capability is a benchmark score really measuring?

A multidimensional view of benchmark performance

BenchMIRT builds on item response theory, or IRT, a framework from psychometrics. IRT estimates a test taker’s ability as well as a question’s difficulty and its power to distinguish stronger performers from weaker ones. BenchMIRT extends this idea with multidimensional IRT, allowing the same question to reflect several latent capabilities.

The system was trained on answers from 100 LLMs across 16 benchmarks and more than 34,000 questions. Importantly, the researchers did not label the benchmarks in advance as safety or reasoning evaluations. When the analysis was repeated, it consistently recovered two major dimensions: safety and general reasoning. These dimensions are statistical patterns in model responses, not a complete definition of what a model is capable of.

At both the model and question levels, BenchMIRT estimates ability, difficulty, and discrimination. This makes it possible to ask which capability is most closely associated with a benchmark result and which questions provide the most useful information.

What the analysis reveals

• BBQ contains more than bias signals. The social-bias benchmark includes questions that also require tracking relationships, following the context, and reasoning from the evidence presented. A low score may therefore reflect reasoning difficulty as well as behavior related to social stereotypes.
• WMDP is strongly connected to reasoning. It tests dangerous dual-use knowledge in biology, chemistry, and cybersecurity, while treating refusal or failure to provide that information as the desired outcome. BenchMIRT found that stronger general reasoning was associated with lower WMDP scores, illustrating why knowledge access and safe behavior should not be conflated.
• HarmBench mixes signals across question types. Its standard and contextual harmful prompts aligned more closely with safety, whereas its copyright questions aligned more closely with general reasoning. One overall score can therefore conceal meaningful differences within the same benchmark.

Selecting fewer, more informative questions

BenchMIRT can also rank questions by how well they distinguish models across the relevant dimensions. In the reported experiments, retaining 10% of the questions generally preserved a similar picture of which models were stronger or weaker on the underlying safety or reasoning capability. Retaining 50% often matched the full benchmark more closely.

The model can use learned patterns to predict whether an LLM would answer an unseen question correctly. Its held-out accuracy was 79%, compared with 70% for a simpler baseline that assumed a model would perform on each question roughly as it did on the benchmark overall. This suggests that evaluation may become more targeted without requiring every model to answer every item.

Why it matters—and what it does not prove

BenchMIRT does not show that existing benchmarks are useless. It shows that a benchmark score is a compressed signal and may not correspond to one clean ability. Future evaluations could report separated dimensions, inspect question clusters, and prioritize items with high information value instead of relying exclusively on a total score.

There are also limits. The findings depend on the models, questions, and response patterns included in the analysis. Statistical associations can reveal hidden structure, but they do not replace human review of question wording, scoring rules, or safety objectives. For model comparisons, examining the composition and inferred dimensions of a benchmark may be more informative than reading a single leaderboard number.

Source: Hugging Face Blog