Sauron Puts Evidence Before Verdicts in Agent Evaluation

Introduction

Evaluating a long-horizon agent is more difficult than checking whether a final page, file, or database entry looks correct. Evidence may be distributed across the environment state and the sequence of actions that produced it. A judge that inspects only the final state can miss process failures, while a judge that accepts the agent’s own report may reward an unsupported success claim.

The problem becomes more serious when rubric verdicts are used as reinforcement-learning rewards. A mistaken positive verdict can teach an agent that bypassing checks, stopping early, or reporting success without proof is an effective strategy. The paper Evidence-First Agentic Verification presents Sauron as an alternative: investigate the evidence first, then decide whether a criterion is satisfied.

Key ideas

• Inspect state and trace together. Sauron selects targeted queries over the current environment and the execution history instead of treating either one as a complete source of truth.
• Verify claims at their source. An agent’s statement that an action succeeded is only a hypothesis. Sauron returns to the relevant source and checks whether the state or trace actually supports it.
• Bound the investigation. Bounded queries limit context use. Criteria connected by shared evidence are investigated jointly, reducing repeated work.
• Make decisions auditable. Each verdict includes a receipt describing the evidence behind it, allowing a reviewer to trace why the verifier accepted or rejected a criterion.

Results

Sauron was evaluated on 2,586 human-labeled criteria drawn from 100 personal-assistant episodes. Across all eight tested backbones, it achieved the highest F1 among five verifiers while costing less than Gandalf, the strongest baseline. With GPT-5.6 Sol, Sauron reached 0.96 F1 versus Gandalf’s 0.92, with 23% lower cost. The advantage was larger on criteria concerned with the process rather than only the final outcome.

The authors also built a paired fault-injection suite covering 13 families of failures, including cases intended to simulate reward hacking. Sauron detected 97% of the injected failures and flagged only 2% of clean controls. It outperformed the baselines even when those systems were given the full trace. On TRAIL, a benchmark for detecting and localizing real agent errors, Sauron led both Gandalf and the benchmark’s own judge on all three reported quality metrics.

Why it matters

The main contribution is not simply a higher evaluation score. Sauron treats a verdict as the end of an investigation, not the output of an impressionistic grader. That distinction matters for automated task acceptance, reward construction, and debugging: an evaluation should be able to show which source supports its conclusion.

The evidence is not universal proof of robustness. The experiments focus on personal-assistant episodes, injected faults, and established benchmarks. More open-ended environments, rapidly changing state, and specialized domains may require different query policies or evidence formats. Even so, the work points to an important design principle for agent infrastructure: as agents become better at acting over long horizons, their evaluators must become better at inspecting what actually happened.

Source: Hugging Face Daily Papers