How Much of Harness Evolution’s Gain Is Truly Generalizable?

Introduction

Agent systems can improve without changing the weights of their underlying models. Harness evolution takes advantage of this possibility by automatically modifying the prompts, skills, tools, and orchestration code surrounding a fixed-weight executor. The approach can produce impressive gains on a search set, but a higher benchmark score does not necessarily mean that the agent has acquired broader capabilities.

A paper on arXiv, titled Harness-Delta Attribution: Dissecting Gains From Harness Evolution, proposes a protocol for asking where those gains actually come from. Harness-Delta Attribution, or HDA, separates an evolution gain into three components: overfitting caused by artifacts tailored to the search set, improvement attributable to test-time compute scaling, and a residual component that cannot be explained by either factor. The residual is not proof of generalization; it is an upper bound on the improvement that may remain generalizable.

Key findings

• The evaluation covers 11 executor models and three harness-evolution frameworks.
• Tasks span mathematics, coding, creativity, and agentic capabilities.
• Among 21 settings with statistically significant search-set gains, overfitting accounts for 54% of the gain on average.
• Test-time scaling accounts for a further 14% on average.
• In 14 of those settings, the two factors together explain more than half of the observed improvement.
• Residual gains and held-out transfer vary substantially across tasks, models, and frameworks.

The central message is not that harness optimization is useless. Rather, its benefits need to be measured more carefully. A revised prompt, an extra tool call, or a longer orchestration process may raise a score because it adapts to the benchmark, because it spends more inference-time compute, or because it genuinely improves behavior. These explanations have different implications for deployment cost and reliability.

The study also reports that search-set improvements frequently fail to transfer to held-out tasks. They provide limited evidence that the gains will persist under practical distribution shifts. Moreover, validation-based signals alone are insufficient to prevent overfitting, so a conventional train-validation split may not fully resolve the problem.

Why it matters

HDA offers a more granular way to report progress in agent research. Instead of presenting only the best search-set score, future evaluations could disclose the role of extra computation, performance on held-out tasks, and the portion of improvement explained by benchmark adaptation. This would help distinguish an agent that has become better at a particular test from one that is more dependable across changing tasks.

The findings do not establish that every harness-evolution method lacks value. They do show that statistically significant benchmark gains can be misleading when their sources are not attributed. Bringing cost, transfer, and distribution shift into the same evaluation framework may become essential as agent optimization moves toward practical deployment.

Source: arXiv