LongRCA Bench Asks Not Just Why Agents Fail, but Who and When

Introduction

When a long-horizon agent fails, outcome-based evaluation usually reports only the final result: the task was not completed. That signal is useful, but it does not explain where the decisive mistake entered the execution. Developers still need to inspect a long sequence of plans, tool calls, handoffs, and observations to determine which role was responsible and which earlier step set the failure in motion.

What LongRCA Bench evaluates

LongRCA Bench is designed around that diagnostic problem. It contains 1,140 real failed trajectories from five domains, without artificially injected errors. The median trajectory is 145 steps long, making the benchmark substantially different from evaluations built around short traces. Each example has independently scored human labels for two related but distinct targets:

• Responsible-role attribution: identifying the role, component, or execution participant chiefly responsible for the failure.
• Earliest decisive root-step localization: finding the earliest step that decisively caused the later failure.

Keeping these targets separate matters. An early handoff may introduce a wrong instruction, while a later agent or tool call may visibly expose the problem. Conversely, an early anomaly may be harmless until a later decision turns it into an unrecoverable failure. A single outcome score cannot distinguish these cases.

RCTA and the reported results

The paper also introduces Root-Cause Trajectory Attribution, or RCTA. It is training-free and uses a two-stage strategy. First, it retrieves candidate error steps from summaries of trajectory segments. It then traces those candidates backward to available earlier handoff instructions. This design aims to reduce the search burden imposed by hundreds of recorded steps without requiring additional model training.

Using the same backbone, benchmark instances, and scoring protocol, RCTA obtains 51.1% accuracy for responsible-role attribution and 24.1% exact accuracy for root-step localization. The strongest baseline reaches only 13.2% on the latter metric. The gap highlights an important distinction: identifying the likely party associated with a failure is easier than naming the exact earliest step that made the failure decisive.

Why it matters

The benchmark shifts agent evaluation from asking whether a task succeeded to asking whether a system can explain its own failure. This is especially relevant for multi-role agents whose behavior depends on delegation, tool use, and handoff instructions. Reliability engineering needs more than a failure count; it needs evidence about responsibility, propagation, and the point where recovery was still possible.

At the same time, the reported exact root-step score shows that long-trajectory diagnosis remains an open problem. Better summaries, richer telemetry, clearer handoff records, and evaluation protocols that capture causal dependence may all be needed. With the dataset and leaderboard publicly available, LongRCA Bench offers a common setting for comparing future diagnostic approaches.

Source: Hugging Face Daily Papers