Agent Evolution Is More Than a Higher Score: The Hidden Costs of Reliability, Coverage, and Transfer

Introduction: when a higher score is not enough

Agent harness evolution makes it possible to pursue recursive self-improvement without updating the underlying model weights. A system can change its prompts, tools, control logic, or surrounding workflow and appear to improve over successive rounds. But a central question remains: does a better benchmark score mean the agent has become more reliable, efficient, and transferable?

A new study from researchers associated with the University of Science and Technology of China examines that question beneath the final score. Rather than treating one evaluation result as the whole story, the work tracks what agents gain, what they lose, and how changes behave under different conditions.

What the study evaluates

The researchers study agent evolution from three angles: its observable behavior, its ability to generalize, and the mechanisms responsible for changes. Four evolution methods are tested across multiple harnesses, general-purpose tasks, and coding tasks. The analysis also uses repeated evaluations, controlled agent expansion, component interventions, and replays of admission decisions.

The process is organized around four stages:

• Select: deciding which candidate changes deserve further consideration;
• Analyze: diagnosing why an agent succeeded or failed;
• Mutate: producing changes to prompts, strategies, or harness components;
• Gate: determining which changes are admitted into the next version.

This setup makes it possible to examine evolution as a sequence of decisions rather than as a single before-and-after comparison.

The key finding: capability is being reshaped

All sixteen evolved agents solved tasks they had previously missed. Yet every one of them also lost some tasks it had previously covered. In one general-task setting, reliability increased by 5.2 points while ten previously covered tasks disappeared from the agent’s coverage. The result challenges the idea that a higher aggregate score necessarily represents an unambiguous improvement.

In practice, evolution may redistribute capability rather than simply expand it. An agent can become better on a selected slice of the benchmark while becoming less dependable elsewhere. Most reliability gains also came with higher deployment costs, creating a trade-off that a headline score does not reveal.

The study finds that specific, targeted modifications usually outperform generic reminders. However, the value of an individual change can shift when it is combined with other changes. The agent with the highest score also failed to produce the largest mean improvement in any of the four evaluation pools. In addition, overly strict admission rules may reject useful alternatives instead of preserving a diverse set of promising candidates.

Transfer depends on the setting

Improvements do not transfer uniformly. On the general-task target, coverage usually declined across both harness types and evolution paradigms. On the coding target, coverage increased when the system used an outer-layer-only harness. This contrast suggests that transfer depends on the task family, the harness design, and where an intervention operates within the agent stack.

Why this matters

The paper argues for preservation-aware evaluation. Future agent benchmarks should measure not only newly solved tasks, but also retained coverage, repeated-run reliability, deployment cost, and performance on explicitly defined target settings. Evolution systems also need tested decision rules and clear applicability conditions, rather than assuming that every successful mutation is broadly useful.

As a constructive follow-up, EvoAudit revises the select, analyze, mutate, and gate stages using these diagnostics and achieves the best performance reported in the study. Its broader lesson is methodological: agent self-improvement should be audited as a changing system, not celebrated on the basis of a single rising number.

Source: Hugging Face Daily Papers