R²Orch Trains Agent Orchestrators Through Recursive Self-Play

Introduction

Solving a complex task with an agent often requires more than generating a final answer. The system must decide how to decompose the task, which tools or sub-agents to invoke, how to coordinate intermediate steps, and when to integrate the results. Training such an orchestrator commonly depends on manually designed curricula and a growing collection of auxiliary skills. R²Orch, or Recursive Self-Play Orchestration, proposes a different approach: let the orchestrator choose its own learning problems, improve through self-play, and learn how to make that improvement process more effective.

How the framework works

• Two nested loops: The inner loop uses self-play to improve the orchestrator on tasks selected by a Proposer. The outer loop improves the way the inner loop learns, including task selection and skill management.
• Recursive Learnability: The Proposer is rewarded for choosing tasks whose rollouts continue to produce different outcomes as skills are withdrawn. This is intended to identify tasks that still contain useful learning signals, rather than tasks on which every rollout has already converged to the same result.
• Skill Withdrawal: Once controlled evaluation within self-play indicates that the orchestrator performs equally well without a skill, the outer loop withdraws that skill. The withdrawal is not merely a deployment optimization: it creates a new opportunity for the system to discover failures and learn beyond its previous support.
• A closed improvement cycle: Failures are distilled into skills, skills are internalized into model parameters, and withdrawal leads the Proposer toward new failures. The long-term goal is an orchestrator that can operate with few or zero externally supplied skills.

Reported results and limitations

The paper evaluates R²Orch on twelve datasets spanning question answering, mathematics, code, and interactive tasks. In the zero-skill setting, the supplied material reports that its 9B orchestrator achieves the best result on all twelve benchmarks. Against the strongest baseline, the average score rises from 86.0 to 89.8 in-distribution and from 76.1 to 78.8 out-of-distribution. On AIME 2026, the reported result increases from 74.0 to 81.3. These figures come from the paper’s summary; assessing their broader significance requires examining training budgets, baseline implementations, ablations, and reproducibility in the full paper.

Why it matters

R²Orch reframes curriculum design as a learning problem. The most useful task is not necessarily the hardest fixed task; it may be the one that continues to reveal meaningful differences after an acquired capability is removed. This provides a principled link between exploration, skill internalization, and deployment simplification.

If the approach scales reliably, it could reduce dependence on manually assembled skill libraries and move agent systems from accumulating external capabilities toward absorbing them into model parameters. Important open questions remain, including the computational cost of nested self-play, the stability of withdrawal decisions, and whether the method transfers to more open-ended tasks.

Source: Hugging Face Daily Papers