On-Policy Self-Distillation Without Supervision: Learning from a Model’s Own Consensus

Lead

Post-training large language models often depends on some form of outside signal: ground-truth answers, reward feedback from an environment, or guidance from a stronger model. The paper “On-Policy Self-Distillation without Any Supervision” challenges that assumption. It proposes U-OPSD, an unsupervised form of on-policy self-distillation that uses only the model’s own generations and their internal consistency as the learning signal.

Core ideas

• Why this matters: Existing OPD and OPSD approaches have shown promise for improving LLMs after pretraining. Yet many of them are still not fully “self” distillation because they rely on labels, feedback, or larger teacher models.
• How U-OPSD works: For a given problem, the model samples multiple rollouts. The method then uses majority voting, constrained by a self-consistency threshold, to construct a pseudo-solution. This pseudo-solution is not externally provided; it is derived from the model’s own repeated attempts.
• What gets distilled: Instead of training on every generated response indiscriminately, U-OPSD conditions the model distribution on the pseudo-solution and focuses distillation on completions that disagree with it. The goal is to correct the model at points where its own behavior reveals a confident but inconsistent trajectory.
• Reported results: On five math reasoning benchmarks—AIME24, AIME25, HMMT25, MATH500, and AMC23—U-OPSD improves Qwen3 non-thinking mode by 8.5% at 4B scale and 10.7% at 8B scale over the base model. It also outperforms OPSD by 3.2% and 2.3% on average in those settings. In thinking mode, it remains on par with OPSD, ahead by 0.9% at 4B and level at 8B, while surpassing GRPO by 0.7% and 1.1% respectively.

Significance and impact

The central contribution is not a new external reward or a stronger supervision pipeline, but a way to turn self-consistency into a post-training signal. If a model can generate several candidate solutions and enough of them converge, that consensus can be used as a provisional target. The method then uses the model’s own disagreements as training material.

This is especially relevant for mathematical reasoning, where repeated sampling often exposes both useful consensus and unstable solution paths. A label-free approach could reduce the cost of post-training and make improvement loops easier to scale across models and training settings.

There are also clear limitations to keep in mind. Majority voting is only helpful when the model’s internal distribution contains a reliable signal. If the model repeatedly converges on the same wrong answer, the pseudo-solution may reinforce that error. U-OPSD should therefore be understood as a way to refine and reorganize existing capability, not as a guarantee of discovering knowledge the model does not possess.

Overall, the paper offers a compelling direction for self-improving LLMs: use the model’s own agreement and disagreement as a supervision substitute, and spend training effort precisely where the model’s outputs conflict with its emerging consensus.

Source: Hugging Face Daily Papers