When Agents Slow Down: Measuring Test-Time Strategy with Elo per Token

Introduction

Once a language model becomes an agent that can revise its answer, use tools, explore alternatives, and decide when to stop, evaluating test-time compute becomes much harder. A final score at the end of a run says little about how efficiently the system used its intermediate steps. Two agents may reach the same result while following very different search trajectories, and a larger token budget may eventually produce very little additional progress.

A paper featured by Hugging Face Daily Papers introduces Elo-per-token analysis to study this problem. The goal is to describe how agent performance changes throughout a long trajectory and to identify when extra test-time computation starts yielding diminishing returns.

Turning intermediate progress into an Elo curve

The researchers focus on open-ended tasks that provide continuous scores for intermediate submissions. At each token budget, they record the best solution discovered so far instead of looking only at the final submission. This makes improvements from revision, tool use, and alternative exploration visible over time.

Raw task scores cannot be compared directly because different benchmarks use different scales. The proposed method therefore uses a Bradley-Terry model to aggregate within-task orderings into Elo ratings that can be compared across tasks. The result is a compute-versus-Elo curve rather than a single end-point metric.

The main observations are:

• Independent sampling provides a theoretically characterized reference whose Elo grows approximately linearly with the logarithm of compute.
• Agents can initially convert tokens into Elo faster than independent sampling, suggesting that revision, tool use, and structured search provide a real early advantage.
• As the per-session budget grows, agents’ marginal gains diminish and can eventually fall below the independent-sampling reference.
• The paper defines a scaling inflection point as the budget where an agent’s marginal Elo gain matches the independent-sampling baseline.

Experiments and human comparison

The analysis is applied to four general-purpose agents across four open-ended benchmarks, with sessions reaching as much as 100 million tokens. The authors also examine three feedback-driven LLM optimization harnesses in controlled, single-task interventions. Together, these settings cover both general agents and more constrained optimization systems.

The study adds a human reference using strong historical contestants on shared AtCoder Heuristic Contest tasks. Those contestants continue to improve superlinearly over contest time. The comparison does not imply that humans always outperform agents. Instead, it offers evidence that people can keep learning from feedback and substantially change their strategies, while current agents often slow down after an initial period of efficient progress.

Why it matters

The value of this work is methodological as much as empirical. It reframes test-time scaling from “give the model more tokens” into a question of marginal efficiency. Evaluations can now ask not only whether an agent succeeds, but also how much each additional token contributes and when the current strategy stops paying off.

For system builders, the inflection point may inform stopping rules, strategy switching, parallel sampling, or feedback-driven optimization. For benchmark designers, Elo-per-token offers a way to compare agents across tasks with incompatible score scales. Most importantly, an agent slowing down may not simply need a larger budget. It may need better feedback use, continual learning, or the ability to reorganize its search policy rather than repeatedly extending the same trajectory.

Source: Hugging Face Daily Papers