Why AI Agent Monitors Should Be Judged by Decision Value, Not Accuracy

Introduction

When an AI agent encounters a suspected failure during a long-horizon task, a monitor may stop the run and launch a fresh attempt. It is tempting to evaluate that monitor with familiar classification metrics: accuracy, true-positive rate, and false-positive rate. A paper featured by Hugging Face Daily Papers argues that these metrics do not answer the operational question that matters most: does acting on the alarm improve the chance of completing the task?

Accuracy does not determine intervention value

The paper presents an analytical example under two simplifying assumptions: the replacement is an independent clean restart, and monitoring adds no overhead. Even with a fixed 90% accuracy, the resulting change in task success can range from a five-percentage-point loss to a 35-point gain.

The variable driving that swing is the probability that the fresh attempt will finish within the remaining budget. If the new attempt is unlikely to succeed, a false alarm can discard an original run that still had a viable path. If restarting substantially improves the odds, an imperfect monitor can nevertheless be valuable. The correct object of evaluation is therefore the full alarm-to-action-to-outcome chain, not the alarm classifier in isolation.

Two tools for decision-oriented evaluation

• Decision sufficiency: Drawing on Blackwell’s comparison of experiments, the paper proposes a criterion for binary-action settings. A progress representation or compressed context is sufficient when it preserves the information needed to choose optimally between continuing and restarting. This focuses on action quality rather than faithful reconstruction of the entire trace.
• Finite-sample decision audit: The authors use simultaneous binomial intervals over existing agent traces. Without additional model calls, the audit can support a preference for restart or continue under stated sampling assumptions. When the evidence is inadequate, it returns “abstain” instead of forcing a decision.

What the empirical study suggests

The study covers 194 externally evaluated model executions on 19 selected SymPy issues from SWE-bench Verified, with a retrospective audit of 130 traces. An apparent 2.29-point restart gain was traced to a single surviving failed attempt. On a second model, restarting produced a 42.29-point loss. The initial positive result also failed to obtain positive prospective replication.

Why this matters

For developers of coding agents, recovery policies should be evaluated with the economics of intervention in view: the probability of success after replacement, the remaining time or token budget, and the cost of interrupting a potentially recoverable run. An audit that can abstain is also important. It prevents sparse or noisy traces from being converted into confident automation rules. The broader lesson is simple: a monitor is not useful because it recognizes failure accurately; it is useful only when its recommended action improves outcomes.

Source: Hugging Face Daily Papers