Why Autonomous Research Agents Fail: Lessons from 800 Trajectories

Introduction

Large language models are increasingly being asked to do more than retrieve papers or draft text. With the right tools and scaffolding, an agent can move from an initial idea through literature search, experiment execution, analysis, writing, and review. This end-to-end setting is often called AutoResearch. It also creates a problem that ordinary answer-based benchmarks are poorly equipped to diagnose: a polished paper can conceal a broken research process.

The paper How Do Agents Fail on AutoResearch presents AutoResearchEval as an attempt to make those failures visible. Instead of asking only whether the final report looks good, the evaluation follows the path that produced it and examines the artifacts left along the way.

A benchmark built around complete trajectories

The benchmark contains 100 real-world tasks grounded in published frontier research. They span seven scientific domains and cover the research lifecycle, including ideation, retrieval, execution, analysis, writing, and review. Eight model–harness combinations were evaluated, producing 800 agent trajectories.

The unit of analysis is not just the final answer. Researchers inspect trajectory logs, source code, run directories, data, and reports. This makes it possible to detect contradictions that a surface-level score can miss: a number in the abstract that does not match the table on disk, a method section describing code that was never run, or a claim that cannot be connected to any experiment.

For the 70 open-ended tasks without a conventional gold answer, the evaluation emphasizes process rigor and internal consistency. The judging pipeline was calibrated against human experts and reported agreement of 0.83. In other words, the benchmark does not need an external answer for every research question if the agent’s own artifacts provide evidence that its account does—or does not—hold together.

What the failure taxonomy reveals

The authors organize their observations into the AutoResearch Failure Taxonomy, or ARFT, with 45 empirically grounded patterns. Several examples illustrate why process-level evaluation matters:

• Anchoring on a failed approach: an agent repeats the same tool call or research route instead of pivoting when evidence shows that it is not working.
• Local optimization and premature stopping: a system completes an attractive subtask without checking whether the overall research objective has been met.
• Right result for the wrong reason: an answer may look correct while its reasoning, experiment, or evidence does not support it.
• Result hallucination and untraceable claims: reported values or conclusions cannot be found in the run artifacts, or the paper describes an implementation that never existed.
• Cascading errors and goal drift: an unchecked early mistake gradually redirects later retrieval, execution, analysis, and writing.

Although these failures appear at different stages, the paper identifies a common underlying weakness: agents do not reliably compare what they produced with what they actually found or ran. They also rarely pause to question a plan, revise a conclusion, or restart after detecting a contradiction. The missing capability is therefore described as a metacognitive loop—checking, doubting, correcting, and replanning within the task itself.

An important limitation

The evaluation is not a complete observability solution. The authors acknowledge that gradual context-window degradation is not isolated as its own mechanism. The benchmark can identify nearby effects such as error propagation and goal drift, but it does not fully observe the per-step context assembled by the harness or how the model uses that context. As a result, some production failures caused by accumulated irrelevant information may be undercounted.

The broader lesson is that scaling an agent to longer research workflows is not enough. A dependable system needs explicit verification checkpoints: compare prose with stored results, compare claimed methods with executed code, and test conclusions against the evidence before moving on. AutoResearchEval shifts attention from a single final score to auditable trajectories and artifact-level attribution. For autonomous science, the next advance may depend less on making agents generate papers faster than on making them recognize, before submission, which parts of their own research still do not stand up.