Harness-of-Harness: Turning Coding Agents into Iterative Software Developers

Introduction

Generating a working code fragment is no longer the only challenge for LLM-based coding agents. A harder problem is maintaining progress over many unattended sessions: fixing regressions, adding useful capabilities, preserving project context, and eventually producing software that people can use. The paper introduces Harness-of-Harness, or HoH, as a framework for this longer-horizon setting. HoH does not replace the underlying language model and is not presented as a new coding agent. Instead, it operates on existing coding-agent harnesses and manages how their executions are organized over time.

How HoH works

The framework structures development as repeated planning, coding, and testing loops, with independent evaluation between iterations. Its main ideas are:

• Combine repair with growth. Each cycle should address existing problems while also moving the product toward additional capabilities, rather than becoming a sequence of emergency fixes.
• Use small, verifiable increments. Narrow deliverables make progress easier to inspect and reduce the risk of losing the original objective during long autonomous runs.
• Separate implementation testing from evaluation. Tests used while writing code provide development feedback, while an independent evaluation checks whether the delivered result actually meets the intended requirements.
• Constrain outputs instead of prescribing workflows. HoH focuses on what must be verifiably delivered, leaving the agent flexibility in how it plans and implements the work.
• Expose resources progressively and favor reuse. Deliverables, role-specific tools, and skills become available as needed, while existing work is reused instead of recreated.
• Maintain versioned history. Recording project states supports comparison, recovery, and further improvement across iterations.

The paper evaluates HoH on GameCraft-Bench, FrontierSWE, and ProgramBench with three harness-model pairings: Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3. The reported results show that HoH outperforms the corresponding standalone harnesses after three iterations, with an average relative gain of 52.25% and a maximum gain of 82.86%. A separate deployment ran for more than 70 iterations and produced a first-person-shooter game with a coherent storyline, implemented core mechanics, a human-playable experience, polished visuals, and integrated audio.

Why it matters

The central contribution is not simply a higher score on a single pass. HoH treats long-running software development as a problem of control, feedback, and state management. For coding agents, these system-level mechanisms may be as important as raw code-generation ability. The results also suggest that progress may come not only from larger models, but from better orchestration of task decomposition, validation, resource access, and project history.

The available material is still an abstract-level description, so it does not establish how the framework behaves across every project size, evaluation regime, or compute budget. A multi-day demonstration also does not mean that arbitrary software can be built without supervision. Further study should examine recovery from repeated failures, long-term code quality, test coverage, and the relationship between iteration count and cost.

Source: Hugging Face Daily Papers