EarlyEval Cuts Agent Evaluation Costs by Predicting Outcomes Early

Introduction

Evaluating an LLM agent is more demanding than checking a single answer. Coding agents, terminal agents, and tool-using systems may need many rounds of reasoning, action, observation, and revision before a benchmark can determine the final result. A single run of a frontier model over an agentic benchmark can therefore cost hundreds or even thousands of dollars. Repeating that process during development makes evaluation a major engineering expense.

Existing efficiency efforts often focus on reducing the number of benchmark tasks, for example through benchmark distillation. EarlyEval targets a different source of cost: the execution length of each individual task. Its central question is simple: if an agent’s eventual outcome is already becoming clear, why continue paying for every remaining step?

How EarlyEval works

The method assumes that intermediate trajectories contain useful evidence about the final result. A sequence of inconsistent tool calls, a solution that diverges from a reference path, or a coherent implementation followed by verification may provide early clues about failure or success. These signals are not treated as a single hand-written rule. Instead, EarlyEval extracts behavioral, textual, and reference-solution features at each stage of execution.

It then trains two lightweight LightGBM classifiers. One estimates the probability of success, while the other estimates the probability of failure. After calibrating their confidence thresholds, the evaluation runner stops a task as soon as either classifier crosses its threshold. The additional computation is intended to be negligible compared with another call to the evaluated agent.

Results and limitations

The experiments cover SWE-bench Verified, TerminalBench, and Toolathlon. Across these benchmarks, EarlyEval eliminates 13%–26% of agent steps. Depending on the setting, it reduces input tokens by up to 44.1% and output tokens by up to 29.4%. Prediction accuracy ranges from 89% to 97%, while per-agent resolve rates shift by only about one to two percentage points on average.

The results suggest that early stopping can save substantial evaluation resources without simply applying a crude fixed-length cutoff. Calibration is important, however. A false failure prediction could stop a task that would eventually succeed, while a false success prediction could end an unsuccessful trajectory too early. Threshold selection and feature robustness may also become difficult when the evaluated model, benchmark, or tool policy changes.

Why it matters

EarlyEval complements task-level benchmark reduction. Rather than removing test cases, it tries to shorten cases whose outcomes have become sufficiently predictable, reserving full execution for uncertain examples. This is particularly relevant for teams running frequent regression tests while developing agents, where even a modest per-task saving can accumulate across many experiments.

The broader contribution is to treat agent evaluation as a dynamic resource-allocation problem. Evaluation systems can use trajectory evidence to decide how much additional computation a task deserves instead of assigning every case the same budget. EarlyEval is not a replacement for full execution: its safety depends on calibrated predictors and careful monitoring of outcome distortion. Still, it offers a lightweight, practical direction for making agent benchmarks more affordable.

Source: Hugging Face Daily Papers