Repo-To-Skill Turns GitHub Repositories Into AI Research Skills

Introduction

An AI agent may understand a paper and still fail to reproduce its method. The missing ingredient is often not theoretical knowledge, but operational know-how: how to prepare data, configure a repository, launch training, interpret evaluation results, and recover from common failures. Such experience is distributed across GitHub code, scripts, and documentation written for human engineers. It is too large to load wholesale into a task context and is rarely packaged for reuse.

The paper Repo-To-Skill treats this know-how as a missing layer in autonomous machine-learning research. It presents DisCo, a research agent that both creates skills and applies them during research. The central idea is to compress repository-level practices into small, verified units that can be selected when a task requires them.

Key points

• A layer between methods and execution: Existing research agents may combine a model backbone with planning, execution, memory, and verification. DisCo adds operational knowledge that helps translate an abstract method into a working procedure.
• Two complementary distillation modes: Task-agnostic distillation extracts broadly reusable abilities from widely used repositories. Task-oriented distillation creates the skills needed for a specific research problem. The first supports a shared library; the second supports adaptation at runtime.
• A large open skill library: AREX-Skill Library contains more than 5,000 verified skills distilled from 1,000 widely used machine-learning repositories. They are organized across 20 areas and 178 capability families.
• Benchmark evidence: With the GPT-5.5 backbone, research harness, and downstream execution budget held fixed, the skill-equipped agent reported higher scores on four benchmarks. The abstract lists gains of 134.3% on MLE-bench, 34.4% on PaperBench, 9.2% on FrontierCS, and 14.0% on PassNet.

Why it matters

The broader contribution is a different way to give agents access to expertise. Instead of placing entire repositories or long documents into context, a system can extract procedures with their conditions, steps, and validation signals, then retrieve only the relevant units. This may reduce repeated exploration and make engineering experience transferable across research tasks.

The approach also creates an important maintenance problem. Repositories change, dependencies become incompatible, and recommended practices evolve. A skill that was once correct can quietly become stale, so a useful library needs continuous checking and version awareness. Skill generation itself also requires safeguards: compressing an incorrect procedure could make the error easier for many agents to reproduce.

For that reason, future evaluations should look beyond one-off task scores. Transferability, freshness, interpretability, and clearly defined failure boundaries will matter as much as raw benchmark improvements. Repo-To-Skill ultimately frames the open-source ecosystem as more than a collection of documents: repositories can become structured, callable, and verifiable sources of operational knowledge for AI agents.

Source: Hugging Face Daily Papers