AutoEnvScaling Lets Terminal Agents Build Their Own Training Environments

Introduction

Terminal agents can operate software through command-line tools, solve engineering problems, and support parts of scientific workflows. Training such systems with reinforcement learning, however, requires more than a collection of prompts. Each task needs an executable environment, a clear objective, a reference solution, and reliable tests that can verify the result.

Most of this infrastructure is still authored by human researchers. That creates a practical bottleneck: the supply of training environments grows at roughly the speed of the research team. AutoEnvScaling addresses the problem by treating environment construction itself as a terminal task.

How the system works

• A proposer builds tasks. Given rollouts from the solver being trained, a proposer agent creates complete Harbor environments, including a Docker image, instructions, a solution, and tests.
• The proposer can run and debug its work. Instead of producing a task description in isolation, the agent interacts with the terminal to execute the environment, inspect failures, and repair issues before submission.
• Solver behavior closes the loop. New solver rollouts provide evidence about which tasks are too easy, too hard, ambiguous, or otherwise useful for the next generation round.
• The roles can be combined. In a recursive self-improvement setting, one model can act as both proposer and solver, allowing environment generation and task solving to evolve together without humans writing every new task by hand.

Reported results

The paper’s abstract states that AutoEnvScaling raises the valid-environment rate by up to 25 times over prompt-only generation, while reducing the cost per valid environment by as much as 75%. In one experiment, training Qwen3.5-35B-A3B on environments generated from GPT-5.6-sol improved its Terminal-Bench 2.1 score by 4.3 points compared with a matched Tmax baseline.

A second experiment tested a more self-contained loop. Qwen3.6-35B-A3B served in both the proposer and solver roles, gaining 12.2 points on Terminal-Bench 2.1 and increasing its Terminal-Bench 4.0 score from 1.6 to 3.2. These figures are reported by the authors and should be read as results within the paper’s experimental setup, rather than as a general guarantee for every agent or benchmark.

Why it matters

The important idea is not simply automated task writing. Terminal environments contain operational details that are difficult to capture in prose: package dependencies, hidden state, tool behavior, file-system effects, and the exact conditions under which a result should pass. Giving the proposer access to a terminal lets it test the environment as an executable artifact, which is closer to the workflow followed by human environment designers.

The reported pipeline also extends beyond software engineering to computer use, professional work, and GPU kernel programming. This suggests that the method may be useful wherever tasks can be packaged with executable checks.

Open questions

Automation does not eliminate quality concerns. A proposer could generate repetitive tasks, exploit weak tests, or produce environments whose difficulty is poorly matched to the solver. The long-term value of the approach will therefore depend on more than valid-environment counts and benchmark scores. Diversity, test robustness, transfer to unseen tasks, and resistance to reward-hacking behavior remain important evaluation targets.

AutoEnvScaling reframes the reinforcement-learning data bottleneck as an engineering task that agents may be able to perform themselves. If the feedback loop remains reliable, training terminal agents could move from manually expanding a task library toward continuously evolving environments.

Source: Hugging Face Daily Papers