Does RL Teach LLMs New Reasoning? The Case of Thinking Leakage

Introduction

Reinforcement learning is widely treated as a way to give language models stronger reasoning abilities, particularly on mathematical and logical tasks. A new paper featured by Hugging Face Daily Papers proposes a more cautious interpretation. When a model performs better after reinforcement learning, the improvement may not always represent the discovery of a new reasoning strategy. In some cases, the training may simply bring back capabilities that were already present in the original model.

The study focuses on Qwen3, a hybrid reasoning family that can switch between THINK and NO-THINK modes through a small number of chat-template tokens. A common post-training recipe runs reinforcement learning in NO-THINK mode and reports better reasoning accuracy. The authors call the possible mechanism behind part of this improvement “thinking leakage.”

Key findings

• NO-THINK outputs become more THINK-like. The researchers applied GRPO to Qwen3-1.7B, 4B, and 8B. As accuracy increased, responses generally became longer and more reflective. The original model also judged these outputs as increasingly similar to its own THINK-mode behavior.
• More RL approaches, but does not surpass, THINK performance. When the researchers increased the scale of the RL update, the trained models displayed more THINK-related behavior and approached the accuracy of the original THINK mode. However, the reported results did not show them exceeding that mode, which challenges the claim that entirely new reasoning ability had been learned.
• The evidence is causal, not only correlational. In Qwen3-8B, the researchers intervened on a low-dimensional activation subspace associated with THINK behavior. Replacing the RL-trained model’s values with those from the original model removed most of the RL gain; performing the reverse exchange transferred a considerable portion of the gain to the original model. The study also traces the RL-induced shift through attention heads that read the empty think block.
• GRPO may favor the behavior by design. Responses receiving favorable advantage-weighted updates tended to look more like THINK-mode responses. A gradient step derived from this signal likewise moved the model toward more reflective behavior.

Why it matters

The paper does not claim that reinforcement learning is useless. Instead, it calls for a more precise account of what post-training changes. In a hybrid model with an explicit reasoning switch, NO-THINK reinforcement learning may not build a reasoning mechanism from scratch. It may partially restore the model’s original THINK behavior through changes in weights, activations, and attention pathways.

This distinction matters for evaluating post-training. A before-and-after accuracy comparison cannot by itself reveal whether a model learned a new strategy, adapted better to a task, or merely reactivated a suppressed capability. Future evaluations may need mode-matched baselines, activation interventions, and causal tests that separate acquiring a capability from calling on an existing one.

For developers, the result is also a practical reminder: the value of an RL recipe should be assessed together with output length, mode behavior, and inference cost, rather than by final accuracy alone. Understanding where a gain comes from can be as important as measuring how large it is.

Source: Hugging Face Daily Papers