Rethinking Verification: Why Agent Harnesses Need Independent Review

Introduction

Tool-using language models rarely solve complex tasks in a single response. They inspect files, run commands, observe outputs, revise plans, and eventually decide that a task is complete. This makes verification a central part of an agent system rather than an optional final check. Yet agent builders still have limited empirical guidance on when verification should happen, who should perform it, and how feedback should trigger a revision.

The paper Rethinking Verification in Agent Harnesses studies these questions by treating the harness—the layer that orchestrates prompts, tools, observations, and control flow—as a test-time scaling mechanism.

Four places to verify an agent

The authors compare 17 verification designs across four stages of the agent lifecycle:

• Initial instructions: ask the agent from the outset to inspect and validate its own work.
• Mid-turn reminders: insert prompts during execution to redirect attention toward progress and potential errors.
• Completion gates: require a test, check, or other condition before accepting the agent’s claim that the task is finished.
• Independent review: send the result to a separate critic, then return the critique to the acting agent for revision.

The designs are evaluated with seven frontier coding models on Terminal-Bench and SWE-bench Verified. The study also introduces VisualBench, where experts judge visual and interactive artifacts. This broader evaluation matters because code correctness and the quality of an interactive artifact are not always captured by the same checks.

Independent review stands out

Initial instructions and execution-time reminders can help, but their average gains on the coding benchmarks are modest and model-dependent. They do not substantially increase average output token use, yet they also do not reliably prevent the kinds of omissions that appear in longer, tool-driven tasks.

Completion gates can deliver larger improvements, but they may trigger repeated tests, extra execution, or additional repair attempts. As a result, their accuracy gains can come with a considerable cost increase.

Independent review offers a more favorable balance. On Terminal-Bench, it raises average accuracy by 5.9 to 9.5 percentage points while using roughly 1.13 to 1.25 times the baseline cost. On SWE-bench Verified, the gain is 3.9 to 6.6 points at about 1.00 to 1.04 times baseline cost. On VisualBench, all four independent-review designs receive positive net preferences over the baseline, ranging from 13.2 to 23.8 points and outperforming the evaluated lower-level designs.

The study also reports that more review rounds or additional critics can improve average Terminal-Bench accuracy by up to another 3.5 points over a single review. This suggests that review is not merely a binary feature; its depth and redundancy can become tunable axes of test-time scaling.

Implications for agent engineering

The main lesson is that verification should be designed as a control loop, not reduced to a stronger system prompt. Developers must decide when to intervene, whether the checker should share the executor’s perspective, how critiques are converted into concrete edits, and how much additional computation the task justifies.

For high-value coding workflows, an independent critic may be a more practical first investment than repeatedly adding reminders to the original agent. For cost-sensitive applications, selective use of review rounds could reserve extra computation for uncertain or high-risk outputs.

The findings also broaden how agent systems should be evaluated. Comparing foundation models alone can hide large differences caused by harness design. In practice, the model may determine the potential ceiling, while verification architecture determines how consistently the deployed system reaches it.

Source: Hugging Face Daily Papers