LEGO-RL Connects Native Coding Harnesses to Reinforcement Learning

Introduction

Training a model to solve software tasks involves more than producing a plausible code patch. A coding agent typically works through a long-running harness: it reads a repository, invokes terminals and other tools, runs tests, observes failures, and sometimes compresses its context before continuing. These behaviors make the system useful in practice, but they also create difficulties for policy-gradient reinforcement learning. A crashed environment can distort the outcome signal, an agent may exploit weaknesses in the reward process, and the action probabilities recorded during rollout may no longer match what the trainer expects to optimize.

LEGO-RL is designed to bridge this gap without changing the internal control flow of the coding-agent harness. The authors train the sparse mixture-of-experts model Qwen3.5-35B-A3B with GSPO and evaluate it through three native harnesses.

Key ideas

• In-process proxying for faithful optimization. LEGO-RL places an LLM proxy inside the harness process and records the raw generation stream. This supports token-level alignment and trainer-side log-probability recomputation. The approach is intended to remain reliable even when the harness compacts context or re-serializes messages.
• Sandbox orchestration for dependable execution. The framework provides scalable sandbox management, image caching, and defenses applied at different stages. These mechanisms are aimed at limiting the effect of infrastructure failures and reducing opportunities for reward hacking.
• Built-in observability. A plugin automates validation and monitoring, while a Live UI exposes detailed trajectory diagnostics. Researchers can inspect how tool calls, execution outcomes, and rewards interact during training.
• Consistent gains across harnesses. On SWE-bench Verified, the reported score rises from 64.0% to 70.4% with OpenHands SDK, from 62.4% to 68.2% with Claude Code, and from 57.2% to 66.6% with OpenCode.

Why it matters

The broader lesson is that coding-agent reinforcement learning cannot treat the model policy and the runtime as completely separate layers. In a long-horizon software task, a generated action is only one part of a chain involving context management, tool protocols, execution failures, and test feedback. If training ignores those runtime details, a policy that looks effective in a simplified setup may behave differently when placed inside a production-style harness.

LEGO-RL keeps the harness behavior largely intact and adds the interfaces needed for optimization, reliable execution, and diagnosis. That could make it easier to transfer research models into different coding-agent frameworks and compare how those frameworks affect learning. Still, the supplied material does not establish the separate contribution of each component, the training cost, or generalization beyond the reported model, harnesses, and benchmark. More detailed ablations and independent reproduction will be needed to assess the framework fully.

Source: Hugging Face Daily Papers