Doc2Harness Tests Whether LLMs Can Engineer Coding-Agent Harnesses

Introduction

Most coding-agent evaluations hold the surrounding harness constant and ask whether a model can complete a downstream task. That setup is useful, but it hides an important source of real-world performance. A harness decides how prompts are assembled, how tools are exposed, how files and processes are managed, and how results are fed back into the agent loop. Doc2Harness asks a different question: can a model engineer that surrounding system itself?

How the benchmark works

Doc2Harness separates construction from evaluation in two stages. In the first stage, a build model works inside a fixed builder harness. It receives an interface-only contract skeleton and a development document that consolidates behavior observed across twelve frontier harnesses. The model must turn those materials into a complete candidate harness inside an isolated container.

The candidate is then frozen and moved into a separate execution environment. In the second stage, each test case runs in its own container and is driven end to end by a scripted mock model. This prevents the construction environment from silently helping the candidate during evaluation and makes runs more reproducible.

Scoring is based on four observable channels: the requests sent to the model, the resulting files on disk, the process exit code, and command output. Positive controls are included to prevent implementations from receiving credit through vacuous or superficial behavior.

Results

The study evaluates nineteen frontier models, with three independent builds for each model, across 328 test cases. GPT-6 Astra records the highest pass rate at 53.05%, while the weakest model reaches 27.44%. The gap is roughly twenty-six percentage points, but the broader result is more revealing: even the top system fails nearly half of the cases.

The failures are not presented as simple evidence that models cannot write code. Instead, they cluster around how cross-cutting functionality is divided among modules. A harness may implement individual operations plausibly while still breaking when state, tools, file changes, process control, and model communication have to remain consistent across a long execution chain.

Why it matters

This benchmark broadens the definition of coding-agent capability. A model that succeeds inside a carefully engineered runtime may not be able to design that runtime from a contract and a behavioral specification. As agents move from short interactions to persistent workspaces and multi-step tool use, system architecture becomes part of the intelligence being evaluated.

Doc2Harness is also useful because it turns a previously vague engineering challenge into a repeatable measurement problem. Model developers can use it to identify structural weaknesses in long-horizon execution, while harness designers can examine whether their interfaces, module boundaries, and end-to-end checks are sufficiently explicit. The results suggest that progress in coding agents will depend not only on stronger generation, but also on the ability to organize many interacting behaviors into a dependable software system.

Source: Hugging Face Daily Papers