Why Long-Horizon Agent Failures Require Continual Search

Introduction

A short agent trajectory can often be debugged by inspecting the final few actions. Long-horizon agents are different. They may plan, call tools, revise goals, and interact with an environment across a large number of steps. When the run eventually fails, the visible error may be only the last consequence of an earlier mistake. The practical challenge is therefore not merely to describe what went wrong, but to locate the earliest actionable cause in a massive record.

The attribution problem is a search problem

The paper argues that root-cause attribution should be understood as a search task. Relevant evidence is often sparse, separated across distant actions, and only indirectly connected to the observed failure. A diagnostic system must therefore search beyond the local context surrounding the final error.

Many existing RCA approaches ask an LLM to inspect a trace once and produce a judgment. That setup can work on shorter trajectories, but it creates a specific failure mode on longer ones: premature commitment. The judge encounters a plausible explanation early, accepts it as the working answer, and stops actively looking for evidence that could qualify or disprove it. This also helps explain why repeated analysis of the same run can produce different confident diagnoses. Long traces contain many plausible stories, but plausibility is not the same as causal responsibility.

What Continual Search changes

Continual Search turns diagnosis into an iterative process. Across successive turns, the judge is prompted to challenge its current explanation and continue looking for evidence that remains unexamined or unresolved. The purpose is not simply to ask the same question repeatedly. It is to keep the search open, expose gaps in the current account, and encourage inspection of parts of the trajectory that an initial reading may have skipped.

The authors evaluate the approach on four existing RCA benchmarks. They also identify a limitation in current evaluation: many benchmarks do not represent the scale of execution traces encountered in demanding agent tasks. To address that gap, they introduce MegaRCA-Mix, a testbed containing 50 human-annotated failure trials focused on long-horizon, execution-heavy settings. The supplied abstract reports consistent gains across multiple benchmark suites and model families, but it does not provide detailed numerical results, so the strength of each individual improvement cannot be assessed from the available material.

Why it matters

The work offers a useful reframing for agent reliability. Debugging is not only a classification problem over outcomes; it is also a process of evidence discovery and hypothesis revision. A useful RCA system should make clear which parts of a trace were examined, which assumptions remain unsupported, and why one event is considered more causally important than another.

Continual search may introduce additional inference turns and therefore extra cost. Future evaluations should measure not only attribution accuracy, but also search efficiency, evidence coverage, and whether a diagnosis leads to a concrete intervention. As agent trajectories grow longer, systems that know how to keep searching may be more dependable than systems that simply produce an answer quickly.

Source: Hugging Face Daily Papers