AutoSaddler Turns Agent Failures into Automatic Harness Improvements

Introduction

The reliability of an LLM agent depends on more than the model generating the next action. The surrounding execution harness determines how prompts are assembled, how tools are exposed, how retries are handled, and how context is carried across a long sequence of interactions. A minor weakness in any of these components can compound over time and turn an otherwise solvable task into a failure.

Designing such harnesses is still largely a manual engineering process. Developers must inspect traces, adjust instructions, change tool settings, and revise control logic, often without knowing whether a local fix will generalize. AutoSaddler proposes a more systematic alternative: use agent failures as data for iterative, offline harness optimization.

How the approach works

The framework combines three main ideas:

• Failure-trace diagnosis: Instead of relying only on a final success signal, AutoSaddler examines execution traces to identify where and why an agent went off course.
• Structured patch generation: The harness is treated as code. Candidate updates can target prompts, tool configurations, or control logic, while remaining constrained to the diagnosed issue.
• Validation-based selection: A patch that repairs one trajectory is not automatically accepted. Candidate updates are tested for broader usefulness, reducing the risk of overfitting to a single failure.

This creates an offline loop. A mini-batch of traces provides failure signals, a diagnostic component proposes an explanation, and a patch generator produces candidate modifications. Validation then determines whether the new harness should replace the previous version. The process allows experience from completed runs to become a durable system update rather than a one-time reflection.

Results and lessons

On GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0, AutoSaddler improves performance over the corresponding base harnesses by 9.0, 9.6, and 10.0 percentage points, respectively. The results suggest that significant headroom can remain in the execution layer even when the underlying language model is unchanged.

The ablations highlight three practical principles. Deep debugging is more useful than shallow reflection because it links failure outcomes to specific execution mechanisms. Targeted editing is preferable to unconstrained rewriting because it limits unintended regressions. Finally, update selection must account for generalization across trajectories rather than rewarding a repair that works only on the trace that motivated it.

Why it matters

AutoSaddler frames harnesses as evolving software systems rather than static wrappers around a model. This perspective broadens the optimization surface for agents: improvements can come from better orchestration, tool use, context handling, and recovery behavior, not only from larger models or longer reasoning.

The approach also leaves open important engineering questions. Automatically generated patches need dependable regression tests, and the quality of diagnosis depends on the coverage and reliability of the available traces. Future systems will likely need stronger safeguards for update selection and evaluation across tasks. Still, for coding, terminal, and other long-running agents, turning recurring failures into reusable framework improvements offers a promising path toward more reliable deployment.

Source: Hugging Face Daily Papers