EvoSkill-GUI Gives GUI Agents Skills That Improve During Deployment

Introduction

For GUI agents, the hardest part of a long task is rarely a single click. The real challenge is maintaining a plan while the interface changes underneath it. A pop-up may interrupt the workflow, a page may load later than expected, or a widget may appear in a different location. Any of these events can invalidate a plan that looked correct before execution began.

Existing skill frameworks address this problem by storing reusable procedures. However, those skills are often designed before deployment and treated as fixed artifacts afterward. EvoSkill-GUI starts from a different premise: a GUI skill should be operational knowledge that can be revised when real execution exposes weaknesses.

Core ideas

• Skills are multi-file packages. Each package includes retrieval metadata, executable plans, backup localization methods, failure-recovery rules, accessibility utilities, and recorded failure cases. The structure separates different kinds of knowledge so the agent can use them at the appropriate point in execution.
• The executor can adapt in the rollout. When the current interface diverges from the expected state, the executor performs an immediate revision instead of following the original path blindly.
• A separate critic diagnoses failure. Failed trajectories are reviewed by an isolated critic under strict information constraints. This is intended to make the diagnosis more disciplined and reduce the chance that the evaluator simply relies on information unavailable during execution.
• Updates are restricted and targeted. The executor edits specific skill files through a constrained tool interface. The goal is to preserve useful procedures while adding a concrete fallback or recovery rule rather than rewriting the entire skill library.

Results and significance

The paper evaluates the approach on MobileWorld, AndroidWorld, and OSWorld, covering both mobile and desktop GUI environments. According to the authors, EvoSkill-GUI improves multiple base models without any additional training. The maximum reported gains are 16.2%, 6.0%, and 10.5% on the three benchmarks, respectively.

The broader implication is a shift in how agent memory is designed. A skill library need not remain a collection of deployment-time instructions. It can become a record of operational experience, in which failed attempts contribute new localization strategies, recovery procedures, or warnings for later tasks. This is particularly relevant to software interfaces whose layout and timing cannot be fully captured by a static workflow.

The approach also raises practical questions. A faulty diagnosis could cause an incorrect rule to be stored, while unrestricted edits might allow errors to accumulate across the library. The value of EvoSkill-GUI therefore lies not simply in letting an agent rewrite its skills, but in combining revision with isolation and constrained editing. It offers a training-free route toward agents that become more robust through use rather than only through model updates.

Source: Hugging Face Daily Papers