AgentProofBench Connects Web-Agent Verification to Execution

Introduction

For a web agent, accepting an output is only one part of the problem. A useful verifier must also remain valid when requirements change and must provide a reliable basis for subsequent actions. Many existing evaluations treat acceptance rules and agent execution as separate concerns. That separation makes it difficult to tell whether a verifier that looks correct in an isolated test will actually help an agent complete a task.

AgentProofBench is designed to close this gap. The benchmark freezes an acceptance rule, pairs it with saved web openings, and connects verification tests to later execution traces. It includes 97 tasks organized into 79 families across four web services. Rather than reporting only a single pass rate, it follows the chain from requirement checking to program gating and eventual delivery.

Key findings

• Reuse is evaluated explicitly. Complete-panel and decision-replay tests examine whether a rule can be reused in subsequent settings, rather than simply producing the right answer once. In 48 families, the experiments retained 82 of 96 source programs for DeepSeek Flash and 79 of 96 for DeepSeek Pro.
• Reuse and reconstruction can differ. The paired reuse-versus-rebuild Bland–Altman intervals were reported as -5.47 to 1.57 percentage points for Flash and -10.16 to 0.00 points for Pro. These intervals provide a concrete way to quantify disagreement between applying an existing verifier and rebuilding one.
• Panel errors are linked to delivery failures. Across 22 execution families, 32 of 33 assignments with error-free panels delivered correctly from both saved openings. Only 1 of 11 assignments with panel errors achieved the same result. The paper reports that all reference-defined program-gate regressions and withheld deliveries occurred in the latter group.
• A neutral baseline matters. Under the tested generic feedback setup, neutral continuation produced the highest correct-delivery rate. It is not presented as a universal best policy, but as a practical baseline for estimating the incremental value of a gate.

Why it matters

The central contribution is a distinction between a verifier being correct and a verifier being useful. A rule can pass a static requirement test and still fail to support execution because its panel is wrong, its decision cannot be replayed, or its state handling causes the agent to reject or withhold an otherwise valid delivery. AgentProofBench makes those failure paths observable.

The released instructions, states, programs, reference rules, and trajectories also make the benchmark useful for diagnosis, not just ranking. Researchers can connect an individual verification failure to the execution consequence that follows. This supports a more realistic evaluation agenda for web agents: acceptance rules should be assessed both as requirement-level judgments and as policy-level control signals. In deployment, a verifier is not merely a final scorekeeper; it is a reusable component that can either reduce or amplify execution risk.

Source: Hugging Face Daily Papers