OPD² Reframes On-Policy Distillation Around What Reasoning Tuning Adds

Introduction

Post-training for large language models is no longer limited to reward-model-driven reinforcement learning. A growing line of work uses on-policy distillation, where a student model receives token-level supervision from a stronger teacher while training on its own generated trajectories. NAVER AI Lab’s paper, “On-Policy Delta Distillation,” proposes a refinement to this setup: instead of asking the student to imitate the teacher’s full distribution, teach it the difference between the reasoning-tuned teacher and the teacher’s original base model.

The method is called OPD², short for On-Policy Delta Distillation. Its central idea is simple but meaningful: a reasoning-tuned model contains both general language-modeling behavior inherited from its base model and new behavior induced by reasoning-oriented tuning. If the goal is to transfer reasoning ability, the most relevant supervision may not be the teacher distribution itself, but the delta introduced by that tuning process.

Key Points

• A new distillation reward: OPD² introduces the delta signal, defined as the difference between a reasoning-tuned teacher and its base model before reasoning instruction tuning.
• More focused capability transfer: By subtracting the base model’s contribution, the method aims to highlight the changes associated with reasoning improvements.
• Built for on-policy distillation: The approach remains within the on-policy distillation framework, but changes the reward signal used to guide token-level learning.
• Tested across reasoning domains: The paper reports improvements over conventional on-policy distillation on mathematics, science, and code-reasoning benchmarks.
• Consistency across model settings: According to the provided material, the gains hold across multiple Qwen3 model sizes, in both thinking and non-thinking modes, and also extend beyond Qwen3 to Gemma 4.

Why the Delta Matters

Conventional teacher distillation can be powerful, but it also carries a risk: the student may learn many aspects of the teacher that are not directly related to the target capability. A teacher model’s output distribution reflects broad language fluency, factual priors, instruction-following style, formatting habits, and reasoning behavior all at once. OPD² attempts to filter this signal by comparing the teacher with its own earlier base version.

For reasoning models, this comparison is especially natural. The capabilities added during reasoning tuning may show up as different token preferences, more structured intermediate steps, better problem decomposition, or different choices in code and scientific reasoning. The delta signal is designed to emphasize those changes rather than the entire behavioral profile of the teacher.

Significance and Impact

If the reported results generalize further, OPD² could become a practical tool for training smaller or more efficient reasoning models with shorter post-training schedules. Rather than copying everything a stronger teacher does, a student can focus on the capability shift that made the teacher better at reasoning.

The work also points to a broader lesson for post-training research: distillation targets do not have to come from a single final model in isolation. They can be derived from contrasts between model states, such as a base model and a reasoning-tuned successor. For teams that maintain model lineages, those historical relationships may become valuable training signals.

The available material does not establish how OPD² behaves in more open-ended workloads, long-context use cases, or production settings. Still, as a targeted redesign of the distillation reward, it offers a clear and testable direction for improving reasoning capability transfer.

Source: Hugging Face Daily Papers