ParetoRSI-Gym Tests Whether AI Agents Can Improve LLMs Across Domains

Introduction

Recursive self-improvement is not simply a matter of asking a language model to produce better answers. A useful RSI system must be able to participate in an engineering loop: find data, curate it, train a successor, inspect the results, and decide what to do next. Many existing evaluations reduce this process to a single benchmark. An agent may select examples from a static pool or distill a stronger model, with success defined by pushing one score upward in one run.

That setup misses an important property of real post-training. Models are expected to improve across several capabilities at once, and progress in one area can come with a regression elsewhere. ParetoRSI-Gym is designed to expose that trade-off.

Key points

• A repeated agentic loop. The agent searches for data, curates a training set, launches training, reads per-domain results, and uses the feedback to plan the next round.
• Controlled experimental conditions. The initial language model, toolset, five domains, and supervised fine-tuning pipeline remain fixed. The main degree of freedom is the data selected by the agent.
• Five evaluation areas. The benchmark covers mathematics, coding, finance, medicine, and science, making a single-score optimization strategy less sufficient.
• Held-out generalization checks. Separate benchmarks test whether the resulting model learned broader capabilities or merely overfit to the proxy scores used by the agent.
• A measurable but uneven improvement. Across three agent systems, three base models, five rounds, and multiple seeds, the strongest configuration—Claude Code with Opus-5—raised the generalization score of its proxy-best checkpoint by 9.2% relative to the starting model, or 2.3 absolute points, averaged equally across the three models and five domains. The largest single-domain relative gain was 63% on medicine with Ministral-3-3B.

Why it matters

ParetoRSI-Gym reframes agent-led model improvement as a sequence of engineering decisions rather than a one-off data-selection exercise. The agent must consider data quality, domain coverage, and conflicting objectives. A higher score in one area is not automatically a successful outcome if it damages another capability.

The use of held-out generalization benchmarks is particularly important. An agent can exploit quirks in the feedback it receives, selecting examples that improve a visible proxy without producing a genuinely more capable model. Independent tests provide a way to separate this kind of optimization from transfer to tasks the agent did not directly optimize.

The analysis also highlights the role of the harness—the surrounding tools, feedback format, and iteration procedure. Its influence can be comparable to that of the model driving it. Moreover, each agent developed a distinct data-selection strategy, suggesting that future comparisons should document not only the underlying model, but also how the system observes results and turns them into training decisions.

The benchmark is not a complete test of recursive self-improvement. It keeps the SFT pipeline and other components fixed, so it focuses on data-driven post-training under controlled conditions. That limitation is also its value: it provides a clearer starting point for studying whether agents can eventually optimize training recipes, evaluation environments, and broader improvement systems themselves.

Source: Hugging Face Daily Papers