Distilled RL brings teacher guidance into reinforcement learning for LLM post-training

Introduction

Post-training has become a key stage for improving large language models, especially when the goal is better reasoning, task adaptation, or alignment. Two common approaches dominate this space. Reinforcement learning optimizes a model against reward signals, but those rewards are often attached to the final output rather than to individual reasoning steps. Online policy distillation, on the other hand, asks a student model to follow a teacher distribution, typically through KL-based matching. The paper “Distilled Reinforcement Learning for LLM Post-training” proposes Distilled RL as a way to combine the strengths of both approaches without inheriting their main weaknesses.

Core ideas

• Standard RL gives sparse guidance. When a model only receives a response-level reward, it is hard to identify which tokens or intermediate steps made the answer better or worse. This credit-assignment problem limits how effectively the model can learn new knowledge.
• OPD can over-imitate the teacher. If the teacher is too similar to the student, it adds little information. If it is very different, matching its logits unconditionally may provide unstable or ineffective supervision. This is one reason OPD is often more natural within a model family.
• Distilled RL uses selective teacher reweighting. Instead of treating the teacher as a target to copy, the method measures the teacher’s relative preference for each token generated by the old student policy. This is done through reverse importance sampling, with clipping to avoid extreme teacher-student probability ratios.
• Negative samples are reset. Teacher reweighting is only applied to positive-advantage trajectories. For responses with non-positive advantage, the weight is set back to one, so the update behaves like ordinary RL rather than learning from a failed trajectory under teacher pressure.
• Sequence-level geometric normalization controls scale bias. The method normalizes token weights within each response, removing sequence-level shifts while preserving the teacher’s relative token preferences.

Results

The authors evaluate Distilled RL using Qwen3-8B-GRPO as the teacher and three student models. On the reported average Pass@1 across ten mathematical reasoning benchmarks, DeepSeek-R1-Distill-Qwen-1.5B reaches 40.00 with Distilled RL, compared with 31.70 for the base model. Qwen3-1.7B reaches 46.37, and Qwen3-4B reaches 58.96. In the table provided by the paper, Distilled RL outperforms standard RL, OPD, and a direct OPD+RL combination for all three students. The authors also report gains in both within-family and cross-family distillation settings, as well as improvements in Pass@1 and Pass@k.

Why it matters

The important shift is conceptual: the teacher model is not used as an unconditional behavior template. Instead, it acts as a fine-grained guide for distributing reward-driven learning signals across tokens. That distinction is meaningful for reasoning-oriented post-training, where a response may succeed or fail because of a small number of key steps.

If this approach generalizes beyond the reported math reasoning setup, it could offer a more controlled way to transfer knowledge from stronger models into smaller or different-family students. The paper’s current evidence is promising, but broader validation across open-ended tasks, safety alignment, and different teacher-student gaps would be necessary before drawing larger conclusions.

Source: Hugging Face Daily Papers