Teaching Language Models to Improve Language Models

Introduction

Language models are increasingly being used to automate parts of AI research, including the development of new language models. In principle, this could create a feedback loop in which a model gains experience proposing improvements and gradually becomes a better research assistant. The practical obstacle, however, is not merely generating plausible ideas. Each proposed change may require a complete training run before researchers can determine whether it works. LM2LM asks whether the ability to propose useful model improvements can itself be trained with reinforcement learning.

How LM2LM works

The system has two main components:

• A proposer model generates candidate improvements and is optimized with reinforcement learning.
• A research world model (RWM) is a frozen language model that reads real experimental records and predicts the likely gain from a new proposal.

A conventional reinforcement-learning setup would need real evaluations for a very large number of proposals. The paper estimates that training the proposer may involve tens of thousands of candidates. If each real evaluation takes roughly ten B200 GPU-hours, collecting rewards could require hundreds of thousands of GPU-hours. LM2LM instead asks the RWM to estimate proposal gains from text-based research records. This turns most reward collection into language-model inference rather than full model training.

The shortcut introduces a central risk: the reward model can become unreliable when optimization shifts the proposer’s output distribution. A proposer may learn to generate ideas that look promising to the RWM but perform poorly in reality. LM2LM addresses this with on-policy calibration. At regular intervals, the system executes proposals that the current policy ranks highly according to the RWM. Their measured results are then appended to the RWM’s context, giving the predictor fresh evidence from the part of the proposal space the policy is actually exploring.

Results and economics

The authors evaluate LM2LM in eight language-model development environments covering autoregressive and diffusion models. On held-out tasks with real execution, the trained Qwen3.8-27B outperformed GPT-6-sol, even when the latter was given 300 real experimental records in context. The paper also reports transfer to an unseen model family, suggesting that the learned behavior is not limited to one development setting.

The cost comparison is a major part of the result. LM2LM uses 200 real runs for calibration, estimated at $40.1k and 343 hours. The in-context-learning baseline based on 300 records requires about $60.0k and 510 hours. Under the paper’s accounting, the LM2LM setup is about 33% cheaper while achieving better performance in the comparison.

Why it matters—and what remains open

LM2LM presents a practical division of labor between prediction and experimentation. Real runs are not eliminated; they are reserved for calibrating the reward model and checking high-value candidates rather than evaluating every proposal. This could make automated research loops more affordable, especially when a single experiment involves substantial training compute.

The approach also highlights an important limitation of research world models. Their predictions cannot be treated as permanent ground truth. As the policy changes, the distribution of ideas changes too, so the predictor must continue receiving real evidence. The method’s success therefore depends on the quality of historical records, the RWM’s generalization ability, and whether calibration experiments reveal the policy’s blind spots. LM2LM is a step toward recursive improvement, not proof that models can independently conduct complete AI research.

Source: Hugging Face Daily Papers