Why Computer-Use Agents Struggle to Recover: ReTrace Makes Recovery Testable

Introduction

Computer-use agents are increasingly capable of operating digital interfaces through fine-grained keyboard and mouse actions. Yet success on a short demonstration does not imply reliability in a long workflow. A single mistaken click can alter application state, enter incorrect data, or create a side effect that cannot be removed by simply refreshing the page. Once that happens, the agent must do more than choose the next action: it must determine how far the environment has diverged from the intended state and find a way back to a state from which the task can continue.

The paper Lost in Recovery examines this underexplored problem. It introduces ReTrace, a scalable framework for studying autonomous, model-based recovery in computer-use agents. Its central idea is to evaluate recovery through goal-oriented action replay rather than through superficial checks of whether the interface appears repaired.

Key ideas

• Recovery is not the same as undoing the last action. In proprietary software and real workflows, resetting the environment may be impossible. Replaying an entire action history can be expensive for long-horizon tasks, while API-based rollback is largely restricted to web settings. A practical agent therefore needs to recover from a state that already contains unintended effects.
• Recovery should be judged by future behavior. ReTrace formalizes Transition-Consistent Recovery. After recovery, the same downstream actions should produce the same outcomes as they would from the target state. This shifts the question from “Did the agent perform a plausible repair?” to “Can the repaired environment support equivalent task execution?”
• Goal-oriented replay exposes hidden damage. The framework injects controlled errors and then compares the recovered state with the desired state through subsequent task actions. This can reveal residual effects and hidden state differences that a visual inspection or a simple success message might miss.
• Latent competence is not the same as recovery competence. The findings suggest that ideal recovery can unlock capabilities the model already possesses for completing the task. However, current computer-use agents often fail to exploit that potential. Their limitation may therefore lie not only in task execution, but also in diagnosing divergence, selecting a recovery strategy, and reconnecting with the original plan.
• Better recovery improves the whole task. Accurate model-based recovery produces measurable end-to-end gains, supporting the view that recovery is a core capability rather than an optional error-handling add-on.

Why it matters

ReTrace offers a more demanding definition of recovery. An agent that removes a warning, rearranges visible elements, or reaches a familiar screen has not necessarily returned to the correct state. Recovery becomes meaningful only when the subsequent workflow behaves as it would from the target state. This criterion gives researchers a clearer basis for comparing models and environments.

The framework also has implications for system design. Error detection, state estimation, recovery planning, and task continuation should be treated as connected components instead of bolting a generic retry mechanism onto an otherwise linear agent. Long-horizon benchmarks may need to report more than final success: they should also measure recovery after controlled failures, consistency of downstream behavior, and the contribution of recovery to overall completion.

The available material does not establish which strategy is best across different applications. Further work will need to distinguish recovery by undoing an action, working around its side effects, or replanning the task. It may also be important to make agents preserve recovery options before taking actions with irreversible consequences. The broader lesson is straightforward: reliable computer-use agents are defined not only by how rarely they fail, but by whether they can return a disrupted task to a usable trajectory.

Source: Hugging Face Daily Papers