HarnessDev Tests Whether LLMs Can Build and Evolve Agent Harnesses

Introduction

An agent’s behavior is determined by more than its model weights. Prompt organization, tool routing, memory, task decomposition, error recovery, and execution control all shape whether the system succeeds. This model-external layer is commonly called the agent harness.

Most agent evaluations hold the harness fixed and compare downstream task results. HarnessDev reverses that perspective. It asks whether an LLM can construct the infrastructure around an agent and then improve that infrastructure through feedback, without changing the underlying model weights.

Evaluating infrastructure, not just answers

HarnessDev contains two stages:

• Creation: Starting with a minimal seed and a small number of cases, the agent builds a complete execution system.
• Evolution: Starting from its own harness, the agent repeatedly revises the system based on downstream execution feedback.

The resulting harnesses are tested on held-out downstream tasks. Evaluation covers both capability, measured by task success, and efficiency, measured by execution-token cost. The reported Creation experiments span six creator LLMs, four domains, and five downstream benchmarks, totaling 2,207 unique instances. Hidden evaluation tasks were withheld during development.

Strong domain dependence

The generated harnesses do not yet match mature human-engineered references everywhere. They remain substantially behind in code, search, and research. These domains often require reliable tool orchestration, information verification, long-horizon planning, and recovery from failed attempts—capabilities that are difficult to obtain from a one-shot generated workflow.

The picture changes in writing and machine-learning experimentation. In those settings, generated harnesses matched or exceeded the selected references. This suggests that harness construction is highly task-dependent. Workflows that can be expressed through language-based planning or adjusted through experiment feedback may be more accessible to current models.

Efficiency adds another layer of complexity. Harnesses can differ greatly in execution-token cost. A system that improves success rates may do so by taking longer reasoning paths, invoking tools more often, or retrying repeatedly. Capability alone is therefore not enough to characterize a useful agent infrastructure.

Self-improvement is not automatically stable

The reported findings also indicate that evolution gains are often unstable and depend on the runtime model. A harness modification that helps one execution model or task distribution may not transfer to another. The model responsible for editing the harness and the model that runs it can interact in ways that make optimization difficult to generalize.

This sets a demanding standard for self-improving agents. Useful evolution requires more than producing additional code or increasingly elaborate prompts. The system must identify bottlenecks, design informative experiments, separate general improvements from lucky outcomes, and balance performance against resource use in a repeatable way.

Why it matters

HarnessDev makes an important engineering layer visible to evaluation. Future agent competition may involve not only larger or stronger models, but also more reliable, economical, and adaptable execution systems. At the same time, the benchmark tempers claims about autonomous agent improvement: current LLMs can build useful harnesses in some domains, yet they remain clearly behind mature human systems in several demanding areas.

Source: Hugging Face Daily Papers