Does On-Policy Distillation Really Distill? The Case for Teacher-Free Adaptation

Introduction

On-policy distillation (OPD) is often presented as a way to combine the dense supervision of token-level training with the exploration benefits of reinforcement learning. The student first generates a trajectory, and a teacher then assigns token-level signals to that trajectory. This appears more informative than the sparse outcome-level advantages used in reinforcement learning with verifiable rewards, or RLVR.

However, the setup contains a basic distribution mismatch. The trajectory comes from the student, not the teacher, so the teacher is evaluating data that is off-policy relative to its own behavior. That raises a central question: are OPD gains really caused by the teacher’s knowledge, or by a simpler effect induced by the training objective?

The paper “Does On-Policy Distillation Really Distill?” investigates this question by analyzing teacher supervision and the locations where learning actually occurs.

Main findings

• Teacher signals are noisy. The authors report substantial noise in teacher supervision during OPD training. Its prevalence increases with teacher scale, challenging the assumption that a larger teacher automatically provides cleaner guidance on student-generated trajectories.
• Students are surprisingly insensitive to that noise. Keeping noisy supervision or removing it leads to comparable final student performance in the reported experiments. This suggests that the teacher’s detailed token-level judgments may not be the main driver of improvement.
• Updates focus on unlikely tokens. Learning is concentrated on tokens with low student log probability. The effect therefore looks less like broad imitation and more like suppressing the tail of the student’s token distribution.
• A teacher may not be required. A single fixed negative advantage can match the performance of teacher-provided advantages, according to the study. If so, part of OPD’s apparent distillation effect can be reproduced without extracting information from a separate model.

From distillation to self-adaptation

These observations motivate On-Policy Self-Adaptation, or OPSA. Instead of querying a teacher, OPSA uses the student’s own uncertainty to determine the strength of each update. High-entropy positions receive stronger learning signals. The method suppresses tail tokens while redistributing probability mass more evenly among head tokens.

On the Qwen3-1.7B baseline, the reported results show a 35.41-point improvement in Avg@32 on AIME24, corresponding to a 263% relative gain. Pass@32 more than doubles across all three reported benchmarks. These results support the paper’s interpretation that structured correction of unlikely outputs can account for a substantial portion of OPD’s gains.

Why it matters

The work does not establish that teachers are universally useless. Rather, it asks researchers to separate the information-transfer component of OPD from the optimization bias created by its token-level updates. If tail-token suppression is the dominant mechanism, expensive teacher inference and elaborate advantage estimation may not always be necessary. A model’s own uncertainty could provide a cheaper training signal.

There are also reasons to remain cautious. The available material consists primarily of the paper summary and partial page information. Details about the noise definition, ablations, task coverage, training stability, and the complete comparison with other methods require inspection of the full paper. OPSA should therefore be viewed as a promising explanation and training direction, not as definitive evidence that teacher-based distillation has no value.

Source: Hugging Face Daily Papers