What Makes a Coding Agent Harness Effective?

Introduction

A coding agent’s software-engineering performance is shaped by more than the underlying language model. The harness determines how the agent plans, invokes tools, preserves useful information, and continues over a long trajectory. Yet many evaluations treat the harness as a single package, making it difficult to tell which component is responsible for a gain or a failure. The paper An Empirical Study of Harness Design for Coding Agents addresses this problem by comparing components inside a controlled, lightweight harness.

Experimental setup

The researchers keep the execution loop fixed and vary three parts: planning, the action space, and context management. They evaluate four models on SWE-Bench Verified and Terminal-Bench 2.1, covering 176 matched settings. The context experiments include five management strategies and four context-window budgets. In addition to success, the study examines cost and trajectory behavior, allowing the authors to connect outcomes with the point at which an agent stops or changes its working pattern.

Main findings

• Context management matters most under tight budgets. When the available window is constrained, agents are more likely to stop because of context overflow before reaching code modification or verification. Management techniques mainly help by keeping the trajectory alive. Their accuracy benefit becomes smaller as the window expands, and they do not substantially change the agent’s underlying behavior.
• Rule-based elision should come before model summarization. Among the tested strategies, removing or eliding low-value content with rules before applying LLM summarization provides the strongest overall efficiency. Making elided content recoverable adds machinery that agents rarely use and does not improve accuracy in these experiments.
• Planning changes from support to optimization. For weaker models, planning acts as an accuracy scaffold: it helps the agent survive long enough to attempt an edit, although this comes at extra cost. For stronger models, planning produces little accuracy change and instead reduces redundant post-edit verification, making the trajectory cheaper.
• The action space should match the model. Predefined tools benefit models with weaker Bash proficiency by reducing dependence on complex shell commands. Models that are comfortable with Bash can work effectively through a Bash-only interface and achieve substantially lower cost, particularly on command-line-oriented tasks. A larger tool catalog is therefore not automatically better.

Why it matters

The broader lesson is that a harness should be designed as a model- and budget-aware layer, not as a universal template. An implementation can first assess whether the model handles Bash reliably, whether the task is command-line-centric, and how restrictive the context budget is. Those factors can guide the choice of planning, predefined tools, and summarization.

The study also suggests that success rate alone is not enough for harness evaluation. Costs, context-overflow failures, stopping points, and trajectory length reveal whether a component genuinely improves problem solving or merely prevents the system from terminating too early. The findings do not identify one best configuration for every agent; instead, they show why future systems may need to switch harness strategies dynamically according to model capability, task structure, and real-time context pressure.

Source: Hugging Face Daily Papers