Keeping Terminal Agents Learning Through Environment Evolution

Introduction

Terminal agents must handle interactive command execution, file operations, software workflows, and other tasks whose success can be checked by an environment. Building such environments is not only a matter of generating prompts: the environment must support interaction and provide reliable verification. The difficulty is that once a model becomes capable, tasks synthesized from scratch may be solved too easily and produce little useful learning signal. A paper from the Tencent Hunyuan team proposes “environment evolution” as a way to keep training challenges aligned with improving agent capabilities.

The core idea

A common approach is to synthesize environments from scratch. This can work for initial training, but strong models may quickly exhaust the challenge offered by simple tasks. More recent co-evolution methods use the agent’s on-policy rollouts to identify weaknesses and create environments near its current learning frontier. The paper argues that this dependency on current-policy data can limit generalization and make it harder to provide learning signals continuously as the model improves.

Environment evolution separates difficulty growth from a purely online generation loop. It incrementally raises difficulty off-policy and schedules the resulting environments generation by generation during training. The authors derive three evolution directions from the multi-turn learning objective, then implement changes along those directions with a loop-engineered multi-agent harness. In this setup, an environment is not treated as a fixed artifact. It becomes a training resource that can be revised as the interaction structure and solution requirements become more demanding.

The method considers more than whether an agent reaches the final answer. Multi-turn action sequences, verification conditions, and the complete path to a solution all matter. This makes the approach resemble curriculum learning for long-horizon tasks: the target remains verifiable, while the reasoning, operations, and coordination required to reach it become progressively more complex.

Experimental evidence

Quantitative rollout experiments with Hy4 preview, Claude Opus 5, and GPT-5.6 Sol showed that evolved environments consistently produced more difficult tasks. The authors then conducted simple long-horizon reinforcement-learning training with Qwen3.6-27B and Qwen3.6-35B-A3B. On Terminal-Bench 2.1, the reported performance gains were 14.4 and 18.0 percentage points, respectively.

These results support the usefulness of environment evolution for training and benchmark performance, but they do not establish that every terminal task will benefit equally. The reliability of verification, the coverage of real failure modes, and the gap between synthesized tasks and practical usage remain important questions for deployment.

Why it matters

The broader contribution is a shift from merely increasing the number of tasks to managing task difficulty over time. In long-horizon reinforcement learning, stronger models require environments that can become more informative rather than remaining static. Generation-based scheduling and multi-agent environment engineering offer a possible foundation for scalable, verifiable interaction training.

More broadly, progress in terminal agents may depend not only on model size or context length, but also on infrastructure that connects environment generation, verification, difficulty control, and training schedules. Environment evolution is a concrete proposal for building that infrastructure around continuously changing learning demands.

Source: Hugging Face Daily Papers