What Outcome RL Really Buys: Better Finishing, Higher Reliability, Little New Reach

Introduction

Outcome reinforcement learning is often described as a way to improve a model’s reasoning ability. But a higher pass@1 does not automatically mean that the model has acquired new knowledge. The gain may come from stopping more effectively, avoiding budget exhaustion, or producing a correct answer more consistently after the reasoning is already complete. What Outcome RL Buys: Finishing, Reliability, Rarely Reach examines these possibilities directly.

The study compares 20 publicly released RL models with the thinking models from which they started. The models range from 1.5B to 32B parameters, and the evaluation covers physics and mathematics problems under fixed token budgets. Multiple samples are used to distinguish persistent behavior from a single lucky solution.

Three sources of change

The authors divide changes in pass@1 into three categories:

• Finishing: whether a model reaches an answer before the token budget expires. An RL model may look better simply because it is better at wrapping up its reasoning.
• Reliability: whether the model is correct once both models have finished. This captures more stable reasoning or answer selection.
• Reach: problems solved by the RL model after the base model failed on all eight attempts, minus problems solved by the base model that the RL model failed on all eight attempts.

Main findings

The largest effect appears when the base model frequently hits the token limit. At 4k and 8k budgets, 82% of the relevant change is attributed to finishing. When the model receives more tokens, the base model closes most of that gap by itself. In these cases, part of the apparent RL gain looks more like improved pacing and stopping behavior than an expansion of underlying knowledge.

When the base model already has enough room to complete its reasoning, the improvement is mainly about reliability. The RL model is more often correct even when both models finish. This is a meaningful benefit, but it should not be confused with broad discovery of previously inaccessible solutions. Under the paper’s definition of reach, the added contribution is at most 2.1 percentage points on any of the four physics and math sets.

The authors also inspected 144 physics problems solved only by the RL model. After resampling or human review, just 3.5% held up. In a random sample, nine out of ten newly observed RL steps confirmed by a human also appeared in samples from the base model. This suggests that many apparent new solutions may have already been available to the base model, with RL mainly increasing their probability or helping them reach completion.

Why it matters

The study is a reminder that pass@1 alone is an incomplete measure of what RL changes. Evaluations should report the token budget, completion behavior, correctness conditional on completion, and coverage across repeated samples. For developers trying to reduce timeouts, training that improves stopping behavior may be highly effective. For developers seeking capabilities absent from the base model, stronger reach-oriented validation is necessary.

This does not make Outcome RL unimportant. It can make a model better at finishing and more dependable. The key distinction is that reliably expressing an existing capability is not the same as acquiring a new one.

Source: Hugging Face Daily Papers