Can Weaker Models Teach Stronger Ones? W2S-OPD Rethinks LLM Distillation

Introduction

Model distillation usually starts from a simple assumption: a stronger teacher is available. That works when a large model is used to supervise a smaller one, but it becomes much less convincing at the frontier, where the student may already be among the strongest models available. The paper “Weak-to-Strong On-Policy Distillation” introduces W2S-OPD, a framework designed for precisely this situation: improving a strong student model using several weaker and cheaper supervision sources.

Key ideas

• On-policy distillation without a stronger teacher: OPD aligns a student with a teacher’s token-level distribution on the student’s own rollouts. W2S-OPD keeps this on-policy setup, but removes the requirement that the teacher itself be stronger than the student.
• A proxy teacher built in logit space: Instead of directly imitating a weak model, the method uses a contrast pair: a positive model and a negative model, both smaller than the student. Their logit difference is treated as a capability direction.
• Multiple forms of contrast: The paper instantiates this idea in three ways: a post-RL expert versus its pre-RL initialization, a larger base model versus a smaller base model, and a small base model with correct hints versus wrong hints.
• Distributional proximity to the student: The extracted direction is added to the student’s own base model, producing a proxy teacher that remains close to the student’s distribution while injecting useful guidance.
• Simple training objective: The student minimizes per-token reverse KL on its own rollouts, learning from this proxy teacher rather than from a conventional stronger model.

Why it matters

The most interesting shift here is conceptual. W2S-OPD treats weak models not as final authorities, but as instruments for revealing directions of improvement. A small RL-tuned model may expose the kind of reasoning structure reinforcement learning encourages; models of different scales may reveal scale-related solving behavior; correct and incorrect hints may highlight instance-level movement toward a solution.

According to the paper summary, W2S-OPD outperforms standard OPD across four math and three code benchmarks, enables the student to surpass the domain teacher, and continues to help even when every supervision source is weaker than the student. The authors also report that different contrast types provide different signals: post-RL and hint contrasts emphasize reasoning frameworks, while scale contrast emphasizes solving procedures.

The available material does not provide full experimental details, model sizes, or cost comparisons, so the practical boundaries still require reading the paper and code. Still, the direction is notable: for frontier LLM training, progress may not always require a more powerful teacher. It may come from extracting useful vectors of capability from weaker models and aligning a strong model to those directions.

Source: Hugging Face Daily Papers