StopLoss Teaches Coding Agents When to Stop

Introduction

Coding agents do not become more likely to succeed simply because they keep running. A trajectory may already be headed toward a failed verification, yet the agent continues editing files, invoking tools and running tests. Having a frontier model watch every step can create a second problem: the monitoring bill may consume much of the savings from stopping early.

StopLoss addresses this problem with a small external monitor. Its name refers to a financial stop-loss order: the system does not repair a bad run or replace the coding agent. Instead, it ends a trajectory when the probability of eventual failure becomes high enough to justify cutting the loss.

How it works

• Learning from outcomes: The researchers use completed coding-agent trajectories together with their final verifier outcomes. Earlier local windows are labeled according to whether the full run eventually passed or failed. The monitor therefore learns patterns associated with eventual failure rather than trying to produce a broad assessment of code quality.
• A compact risk scorer: A language model in the roughly 7B–8B range reads the most recent tool steps and produces a failure score in one forward pass. It does not need to process the entire history or ask a larger model to perform an open-ended review.
• A conservative stopping rule: A threshold calibrated on successful runs converts the score into an abort decision. The monitor can sit outside the executor, so the underlying agent does not need to be retrained or structurally modified.
• Direct judge comparison: The paper compares StopLoss with six general-purpose judges reading identical windows. Experiments cover SWE-bench Verified and a SWE-bench Pro subset, using both GPT and Claude executors.

Results and implications

Outcome supervision turns failure recognition from a signal close to chance at the first check into a useful predictor. Each StopLoss check costs roughly one to two orders of magnitude less than the tested general-purpose judges. On SWE-bench Verified, the method lowers cost per retained solve by 4.5% relative to completing every task while preserving 99% of successful runs. On the more expensive Pro subset, the reduction reaches up to 31%; with a Qwen3-8B backbone, the reported reduction reaches 48%. In all four experimental settings, its cost per retained solve is below that of every judge policy tested.

The broader contribution is a practical division of labor. The executor spends its budget attempting to solve the problem, while a cheaper monitor estimates whether further investment is sensible. For long-horizon coding tasks, this converts failure recognition from expensive open-ended reasoning into a fast risk-scoring operation.

The approach is not a universal failure certificate. Thresholds may need recalibration for a new executor, verifier or task distribution, and errors can either terminate a recoverable run or let a doomed run continue. StopLoss is therefore best viewed as a conservative cost-control layer, not an infallible replacement for verification. The release of its code, models and data should make it easier to test how well outcome-trained monitors transfer across agents and benchmarks.

Source: Hugging Face Daily Papers