WhatWorkedBench Tests Whether AI Agents Understand What Makes Experiments Work

Introduction

An AI research agent must do more than write code and launch experiments. It also needs to determine which change caused an outcome and decide what to measure next. WhatWorkedBench isolates this capability as experimental understanding: the ability to recover how workflow components affect scores from a limited set of observations.

The benchmark asks an agent to inspect workflow code, choose measurements within a budget, and submit a response surface predicting the score for every configuration of component settings. This shifts evaluation away from whether one run succeeds and toward whether the agent can construct a reliable model of experimental effects.

Key findings

• Exhaustive execution supplies a reference space. The authors run configurations on CPUs and measure the effect of changing one component while holding the others fixed. The benchmark spans 36 tasks, 30 data sources, and eight workflow types, with 1,248 configuration records. Its core evaluation combines 4,206 numerical-control records and 108 agent episodes.
• Measurement selection directly affects decision quality. With eight new measurements, pair-effect ridge regression selected an optimum for 15 of 22 sources. On three sources, every effect error stayed within 10% of the score range. The result highlights that choosing informative measurements can matter as much as fitting a more capable predictor.
• Gaussian processes extract more from the same observations. Fitting a Gaussian process to the agent’s existing measurements raised effect recovery from 0.632 to 0.698 in the original Flash cohort and from 0.621 to 0.720 in an additional cohort. Across six completed beat-detection and graph submissions, family-level recovery increased from 0.303 to 0.455.
• Program structure provides useful prior knowledge. In workflows with six binary options, encoding configurations that behave identically in code raised Gaussian-process recovery from 0.248 to 0.462 at 20 new measurements. The finding suggests that code analysis can reduce the effective search space before more experiments are run.

Why it matters

WhatWorkedBench turns a vague expectation—“the agent should understand its experiments”—into a measurable prediction task. An agent may generate plausible code while still being unable to distinguish a real component effect from noise or from a change that happened only in one configuration. That weakness becomes costly when execution budgets are small.

The benchmark also points beyond language-model capability alone. Active measurement selection, numerical inference over a response surface, and detection of behavioral equivalence in programs are all part of effective experimental automation. These capabilities could support comparisons of adaptive experiment strategies and more systematic studies of autonomous coding research.

The benchmark should not be read as a complete measure of scientific discovery. It focuses on recovering component effects and predicting scores. Even so, it offers a concrete foundation for testing whether research agents can learn from experiments rather than merely produce them.

Source: Hugging Face Daily Papers