Can Agent Optimization Compound? A Continual-Learning Test on Terminal-Bench 2.0

Lead

Most agent-optimization results are reported as one-shot improvements: optimize an agent on a fixed benchmark, measure the gain, and treat that gain as a stable property of the method. The problem is that deployed agents rarely live in such a frozen environment. They encounter new failures, new workflows, and new task distributions, and teams often run optimization again as fresh evidence accumulates.

This arXiv paper asks a more operational question: do optimizer-driven gains compound? In other words, after an agent has been optimized once, can it be optimized again on newly arrived tasks without erasing what the first round achieved?

To study this, the authors build a two-phase continual-learning evaluation from hard tasks in Terminal-Bench 2.0. They compare three approaches to agent-harness optimization—GEPA, Meta Harness, and RELAI’s Verifiable Continual Learning, or RELAI-VCL—using identical optimization budgets.

Key points

• Static gains can be misleading. A fixed benchmark can show that an optimizer improves an agent once, but it does not reveal whether the improvement survives future optimization cycles.
• All methods improve in the conventional setup. In the static, single-phase evaluation, GEPA, Meta Harness, and RELAI-VCL all outperform the baseline agent.
• New tasks create a sharp split. Once unseen tasks are introduced, GEPA’s optimized agent transfers below the unoptimized baseline. Meta Harness transfers well, but does not improve further when given a second optimization budget. RELAI-VCL is the only method that both transfers positively and continues improving after the new tasks are folded into the objective.
• Lifelong performance favors RELAI-VCL. The reported lifelong average pass rate is 76.4% for RELAI-VCL, compared with 66.0% for GEPA, 64.6% for Meta Harness, and 58.7% for the baseline.
• Regression control is the central design lesson. The authors’ main observation is that gains compounded only when the optimization loop included mechanisms to control regressions, creating a bias against shortcut solutions that fail to generalize.

Why it matters

For coding agents, terminal agents, and automated workflow systems, continual optimization is not an edge case; it is the normal deployment path. If each new round of tuning improves recent failures while damaging earlier capabilities, short-term benchmark gains may translate into long-term maintenance risk.

The paper therefore shifts attention from single-shot pass rates to lifecycle behavior. A robust agent optimizer should preserve prior gains, transfer to new tasks, and keep improving when new and old objectives are combined. In this study, RELAI-VCL is the only evaluated approach that satisfies all three conditions.

The results should still be read in context: the evaluation uses a specific two-phase setup built from hard Terminal-Bench 2.0 tasks, and broader validation would require more domains and longer horizons. Even so, the framing is important. The real test for agent optimization is not whether an agent can be made better once, but whether it can keep getting better as the world changes.

Source: arXiv