ProgramGym Trains Coding Agents to Reproduce Programs from Scratch

Introduction

Coding agents are increasingly expected to do more than edit an existing repository. A more demanding scenario is program reproduction: the agent studies a reference program, infers requirements that may not be explicitly documented, chooses an architecture, and builds a working implementation over an extended sequence of actions. This setting is closer to software engineering, but it is also much harder to train and evaluate reliably.

ProgramGym addresses this challenge with two connected components: Program-Env, which constructs verifiable training environments, and Program-Agent, which organizes the agent’s development process. The contribution is therefore not only a new agent workflow. It also focuses on the infrastructure needed to generate useful, repeatable feedback during long-horizon coding tasks.

Key points

• Feedback-driven environment construction. Program-Env starts from repositories and builds runtime sandboxes together with black-box tests. It then uses feedback to repair sandbox build failures and improve test quality, aiming to make each environment executable and verifiable.
• Broad systems-language coverage. The reported pipeline produces 1,648 training environments spanning C, C++, Go, and Rust. This gives the benchmark a focus on programs and toolchains that are often more demanding than simple scripting tasks.
• Role-isolated development. Program-Agent separates analysis, implementation, and review. The intended behavior is to let the agent explore and reason about the reference program before committing to an implementation, rather than immediately producing code from a shallow interpretation.
• Revision and recovery. Compilation checks, revision management, and failure recovery are built into the workflow. These mechanisms give the agent explicit opportunities to respond to build errors and refine its solution during a long task.
• Reported performance gains. On ProgramBench, fine-tuning Qwen3.8-27B on trajectories generated with the constructed environments raises the reported score from 34.87% to 42.84%. Program-Agent further reports scores of 55.71% for the base model and 56.25% for the SFT model, exceeding some flagship models under the paper’s setup.

Why it matters

ProgramGym frames program reproduction as a complete engineering loop: observation, inference, design, implementation, compilation, testing, and revision. This is an important distinction from code completion, where the surrounding repository and requirements often provide most of the necessary context. A long-horizon agent needs dependable feedback, and that feedback is only useful when the sandbox runs consistently and the tests probe meaningful behavior.

The environment-building component is consequently as important as the agent architecture. Weak tests can reward superficial or hard-coded solutions, while unstable build environments can make failure signals ambiguous. Program-Env’s repair loop is designed to address both practical problems before the environments are used for training.

The framework also offers a common setting for studying agent workflows. Separating roles may make it easier to examine when an agent fails during exploration, implementation, compilation, or review. At the same time, the available material is limited to the paper’s summary. It does not provide the detailed procedure for generating black-box tests, the distribution of the 1,648 environments, or results broken down by language and task difficulty. The reported numbers should therefore be read as the paper’s aggregate findings rather than a complete characterization of the system.

If the framework scales as intended, coding-agent evaluation could move beyond isolated issue resolution toward the more realistic question of whether an agent can build a runnable program from incomplete information. That shift would make environment quality, automated verification, and recovery planning central parts of future coding-agent research.

Source: Hugging Face Daily Papers