Work2Skill Turns Human GUI Experience into Reusable Agent Skills

Introduction

Computer-use agents are becoming capable of handling increasingly complex graphical interfaces. Their weakness often appears not in basic clicking or typing, but in unfamiliar applications and workflows where success depends on procedural knowledge. A human may know which steps are essential, which decisions depend on earlier results, and how to verify that a task is actually finished. A raw action trace does not necessarily preserve that knowledge.

Work2Skill addresses this gap by treating human GUI activity as experience that can be converted into reusable agent skills. The aim is not simply to replay a sequence of mouse movements and keystrokes. Instead, the system tries to identify the rules and checks that make a workflow transferable to a new instance of a task.

Core components

• WorkRecorder: The recording infrastructure aligns screen video, mouse and keyboard events, and available accessibility-tree information. This gives the induction process multiple views of what happened and what was visible when it happened.
• WorkDemo-350: Using WorkRecorder, the authors assembled 350 human desktop GUI demonstrations. The examples were screened for completeness and successful task execution, providing a more controlled basis for skill induction and evaluation.
• On-demand evidence inspection: The proposed method examines demonstration evidence when needed and extracts workflow rules, dependencies between steps, and criteria for checking completion. This is a more structured target than merely describing an action sequence.
• Cross-model evaluation: Under shared execution settings, Work2Skill achieved the highest scores and success rates among the compared skill-induction methods. Relative to each model’s No Skill baseline, mean scores rose by 4.67–10.54 points on MyPCBench Bounded Action and by 3.11–7.50 points on a 286-task subset of OSWorld-Verified.

Why it matters

The central idea is that reusable GUI competence lies between a rigid script and a vague natural-language instruction. A script can break when a window moves or an input changes, while a generic instruction may omit the dependencies that keep a workflow correct. By representing rules, prerequisites, and completion conditions, a skill can preserve the logic of a human procedure while leaving room for the agent to adapt its concrete interactions to the current interface.

The work also highlights a data-collection challenge. More trajectories alone may not be enough for computer-use agents; demonstrations need synchronized context and reliable success labels. WorkRecorder supplies the capture infrastructure, and WorkDemo-350 provides a screened starting point for studying how experience becomes persistent capability.

The reported results are encouraging, but they do not establish that every induced skill will remain reliable across software versions, layouts, or broader task families. Long-term robustness and scaling remain important questions. Even so, Work2Skill offers a practical framework for moving GUI agents from imitation of actions toward reuse of procedural knowledge.

Source: Hugging Face Daily Papers