RLVR Locks Reasoning at the Entrance, Not in the Computation

Introduction

Reinforcement learning with verifiable rewards, or RLVR, is effective at improving single-sample accuracy. Yet a higher pass@1 can hide a loss of behavioral breadth: after training, a model may repeatedly favor a small set of familiar reasoning routes. If so, drawing more samples at test time produces fewer genuinely different attempts.

A new study asks where this contraction occurs. Does RLVR make some valid solution families inaccessible, or does the model enter them but fail to carry out the remaining computation? Using the Countdown task, the authors argue that the main failure happens at the entrance to a reasoning trajectory rather than inside the computation itself.

Key findings

• Solution coverage contracts. Countdown solutions can be grouped by their first operand and operator, creating discrete entrance families. Across PPO applied to Qwen2.5-3B and GRPO applied to Qwen2.5-3B-Instruct, coverage fell by as much as 67%. Even problems solved at every training checkpoint could lose about half of their covered solution families.
• The largest policy shift is early. Token-likelihood changes before the first arithmetic operation were 11 to 16 times larger than changes during downstream reasoning. RLVR therefore appears to reshape the model’s initial choice of route more strongly than its later ability to compute.
• Latent execution survives. Supplying only an entrance prefix that the model had not selected often restored completion. In the PPO setting, completion for low-access families rose from 0.018 to 0.212. The result indicates that the downstream procedure remains usable once the model is placed on the path.
• Targeted interventions work better than generic prompts. Surface-level prompting did not recover much diversity. Interpolating late-layer parameters with earlier checkpoints increased solution coverage by 37% without reducing pass@1.
• The collapse is not inevitable. Early-step entropy contraction appeared across six math benchmarks and 7B and 14B models, but an SFT baseline retained more than twice the coverage. A staged SFT–DPO–RLVR pipeline also preserved early-step entropy.

Why it matters

The paper turns a broad concern about “RLVR reducing exploration” into a more precise diagnosis. The model does not necessarily forget how to complete alternative solutions; it becomes less likely to initiate them. This distinction matters for both training and evaluation.

Test-time scaling depends on trajectory diversity. If the policy collapses before the first meaningful operation, simply increasing the number of samples may mostly repeat the same entrance decisions. Monitoring early-step entropy and entrance-family coverage could therefore complement pass@1 and other outcome metrics during RLVR training.

The findings also suggest a design direction: preserve or restore diversity specifically at the beginning of reasoning, rather than applying a generic diversity prompt after the policy has already narrowed its choices. Still, the experiments center on Countdown and mathematical benchmarks. Whether the same entrance-locking pattern governs code generation, open-ended reasoning, or multimodal tasks remains an open question.

Source: Hugging Face Daily Papers