RESOURCE2SKILL Turns Human Tutorials into Executable Agent Skills

Introduction

A major obstacle for software agents is not only reasoning, but knowing how to operate real tools. The web already contains a large amount of procedural knowledge in tutorial videos, code repositories, documentation, articles, and polished examples. Most of it, however, is written or recorded for humans rather than packaged as something an agent can directly retrieve, adapt, and execute. RESOURCE2SKILL, introduced by Microsoft Research authors, aims to close that gap by converting multimodal human-created resources into executable agent skills.

Key ideas

• A broader source of skills: Many current skill libraries are hand-written, text-heavy, or summarized from agent traces. RESOURCE2SKILL instead draws from tutorial videos, repositories, articles, and reference artifacts, treating them as complementary sources of procedural knowledge.
• A hierarchical multimodal Skill Wiki: The extracted skills are organized into a structured wiki. Each entry can include text instructions, code, visual examples, metadata, and provenance, making skills easier for agents to inspect, select, and apply.
• Different media contribute different signals: Videos capture the temporal order of operations, interface actions, and visual effects. Code reveals executable tool patterns. Articles and artifacts provide conceptual framing, style, and intent.
• Retrieval and composition at inference time: When facing a task, an agent can retrieve relevant skills from the wiki and compose them. If the existing wiki does not cover the task well enough, the same construction operator can acquire new skills online.
• Results across diverse software domains: The paper reports experiments in PowerPoint, Excel, web generation, Blender, CAD, UI/UX, and music production. Across these seven domains and four model backends, RESOURCE2SKILL improves the average overall score by 11.9 percentage points over no-skill agents, improves performance in all 28 model-domain settings, and outperforms strong agentic harness baselines in 26 of 28 main aggregate settings.

Why it matters

The interesting part of RESOURCE2SKILL is its framing of skills as an interface between foundation models and practical software environments. Instead of relying only on larger models, manually crafted prompts, or bespoke automation scripts, the system suggests that agents can be adapted by systematically harvesting the procedural knowledge humans have already published.

That is especially relevant for creative and authoring software. Building a slide deck, editing a spreadsheet, modeling in Blender, designing an interface, or producing music each involves different workflows, tools, and notions of quality. A purely textual instruction often misses important visual and temporal cues. A multimodal skill format can preserve more of the context an agent needs to reproduce useful behavior.

The approach also raises practical questions. Skill quality will depend on the quality of source materials, the extraction pipeline, and retrieval decisions. Online acquisition may require careful filtering and provenance tracking. Still, RESOURCE2SKILL sketches a compelling direction: many useful agent skills may already exist implicitly across the internet, waiting to be extracted, organized, and made executable.

Source: Hugging Face Daily Papers