Harness or Model? A Controlled Test Challenges Native Coding-Agent Advantage

Introduction

An agentic coding system is more than a language model. Its tools, prompts, task loop, planning behavior, and stopping rules form a harness that turns a chat model into an autonomous software engineer. Vendors generally tune their native harnesses for their own models, encouraging the assumption that the official pairing should solve more tasks. A new arXiv study tests that assumption directly rather than comparing complete products as black boxes.

What the experiment found

• The researchers used a private, contamination-controlled suite of 256 repository and post-cutoff contest tasks.
• For Claude Opus 4.8, the main comparison placed claude-agent-sdk against deepagents. For GPT-5.5, it compared the OpenAI Codex SDK with deepagents. The primary cells used the same 80 tasks per model, while Gemini 3.5 Flash and DeepSeek V3.2 appeared as side cells.
• Of 800 planned runs, 792 were graded by an isolated oracle. Opus scored 48.8% versus 50.0% across the two harnesses, a difference of -1.25 percentage points. GPT-5.5 scored 55.6% versus 54.4%, a difference of +1.25 points. Both confidence intervals included zero, so neither contrast establishes a reliable average winner.
• The Opus average concealed a sharp split. On 61 repository tasks, the native harness trailed by 9.0 points; on 19 contest tasks, it led by 23.7 points. Because the partition was selected after viewing the data, the authors treat this as a hypothesis rather than a final generalization and call for a designed replication.
• Completion and correctness were not interchangeable. Among 81 runs stopped at the wall-clock limit, 22 had already generated patches that passed the grader. Treating every timeout as a simple failure can therefore distort an agent evaluation.

Cost caveats

Using raw per-turn usage and frozen list prices, the neutral harness cost roughly 1.3 to 1.6 times more per solved task than the native setup on Opus, and about 1.2 times more on GPT-5.5. These are observed-usage estimates rather than definitive billing results. Fifty-eight Anthropic-account runs had no usage record; assigning that missing spend to either cell moves the Opus cost ratio between 0.7 and 2.3. The ordering is consequently unresolved. The revision also corrects an earlier manuscript whose cost calculations were affected by a telemetry-semantics defect.

Why it matters

The paper does not show that all harnesses are equivalent. It shows that “native optimization” cannot be assumed without a same-model, task-matched evaluation. Developers should report task strata, timeout policies, completion status, and usage accounting alongside a headline success rate. The authors release the orchestrator, grading oracle, reanalysis code, and derived aggregates, while keeping the tasks private. A stronger conclusion will require preregistered strata and independent replication across more models and harnesses.

Source: arXiv