What RL Really Adds After Pretraining: A Chess-Based Study of Reasoning

Introduction

Reinforcement learning has become a central ingredient in improving the reasoning performance of large language models. Yet it is often studied as a post-training stage in isolation, as if its effects could be separated cleanly from everything that happened during pretraining. The paper Understanding Reasoning from Pretraining to Post-Training challenges that view by asking two practical questions: how do pretraining choices affect the return on RL compute, and what does RL actually do to the model’s policy?

Instead of using a broad and noisy web-scale language corpus, the authors turn to chess. Chess offers structured data, clear rules, and verifiable rewards, making it a useful controlled testbed for reasoning across the full training pipeline.

Key points

• A full pretraining-to-RL pipeline: The study follows a standard LLM-like sequence. Models ranging from 5M to 1B parameters are pretrained on human chess games, supervised fine-tuned on synthetic reasoning traces, and then trained with RL on chess puzzles where rewards can be verified.
• Pretraining loss predicts post-RL performance: At a fixed amount of RL compute, the eventual post-RL score is strongly related to the model’s pretraining loss. This suggests that RL performance is not independent of the foundation built earlier.
• More pretraining tokens improve RL learning speed: The authors report that the slope of RL reward curves improves approximately linearly with the number of pretraining tokens. Better or longer pretraining does not only raise the starting point; it also appears to make RL more efficient.
• RL does more than sharpen SFT behavior: On easy puzzles, RL tends to amplify correct moves that the SFT policy already preferred. On hard puzzles, however, RL can bring out correct moves that were almost absent under the SFT policy, suggesting a broader change than simple confidence boosting.
• The pattern also appears in math: The paper further tests a 1B language model trained on math-domain text and observes a similar trend: longer-pretrained checkpoints reach higher post-RL performance and improve faster during RL.

Why it matters

The main contribution of this work is methodological as much as empirical. By putting pretraining, supervised fine-tuning, and RL into one controlled pipeline, the study offers a more quantitative view of how reasoning capabilities develop. For model builders, the implication is straightforward: RL compute may be most valuable when the pretraining base is strong enough to make use of it.

The findings also refine a common intuition about RL post-training. RL is not always just a way to make the model more confident in what supervised fine-tuning already taught it. In harder settings, it may help expose solutions that were not prominent in the SFT distribution. That makes RL look less like a final polish step and more like a mechanism that can alter the reachable reasoning behavior.

Chess is still a simplified and highly structured domain, so its lessons should not be overgeneralized to all open-ended language reasoning. But as a low-noise experimental environment, it gives researchers a clearer way to study the interface between pretraining and post-training.

Source: Hugging Face Daily Papers