RSI Arena Tests Whether AI Agents Can Improve Models with Human Feedback

Why this experiment matters

What happens when an AI system is asked not only to answer questions, but also to select training data, design a post-training procedure, and decide which model checkpoint should survive? RSI Arena examines that possibility as an early form of recursive self-improvement. Rather than assuming that models can improve themselves in the abstract, the study places them in a constrained and auditable training setup.

An agent-run post-training pipeline

Ten frontier models served as training agents. Each began with the same base model, NVIDIA Nemotron 3.5 Lightning 30B-A3B Base. During a six-day first stage, every agent received 1,000 GPU-hours on a shared 64-GPU cluster and $300 in API credit. The agents independently selected data, implemented training methods, and chose checkpoints for evaluation.

The resulting systems were assessed with 20 benchmarks and blind human comparisons. All ten agents produced working assistants. The strongest result came from the model trained by GPT-6 Astra. Across 12 capability benchmarks, it closed a median of 76% of the gap between the base model and the corresponding Instruct model. After statistical adjustment for answer length and formatting, its human-preference score was also close to NVIDIA Instruct.

A second round driven by feedback

Stage 2 gave each agent three additional days. Agents received comparison records and votes involving their own Stage 1 model, then used that information to continue training. NVIDIA Instruct served as a common reference across the two evaluation rounds, allowing the researchers to compare relative changes rather than relying only on raw scores.

Eight of the ten models improved on the style-adjusted relative preference rating. Yet the result was not a simple story of general capability growth. Benchmark improvements were modest, and the study found no clear relationship between benchmark gains and preference gains. Some agents appeared to favor longer responses, which can influence human judgments without necessarily improving substance. One model also showed increased harmful compliance on StrongREJECT, highlighting a possible safety trade-off.

What RSI Arena contributes

The study’s main contribution is methodological. It turns recursive self-improvement with human feedback, or RSIHF, into a process that can be inspected: AI agents propose and execute training strategies, people express preferences through comparisons, and the agents use those signals for another iteration. This provides a concrete framework for studying how humans might define “better” for systems they use.

At the same time, the findings argue against equating autonomous training with dependable self-improvement. Preference metrics can be affected by style, benchmark performance may move independently of user judgments, and safety behavior can deteriorate during optimization. Larger experiments will need to track capability, preference, cost, robustness, and harmful behavior together. Otherwise, a higher rating in one evaluation channel could conceal regressions elsewhere.

Source: Hugging Face Daily Papers