Weaken to Strengthen: WISD Self-Distills Reasoning Models Without an External Teacher

Introduction

Improving a reasoning model often requires a stronger teacher, labeled solutions, or additional information that can identify which reasoning process is preferable. Those resources are not always available when developers only have unlabeled prompts. A paper from Jet-AI proposes Weakening-Induced On-Policy Self-Distillation, or WISD, as an alternative. Rather than constructing a stronger self-teacher, WISD deliberately creates a weaker version of the original model and uses the difference between the two as a learning signal.

How WISD works

The method starts with an original reasoning model and produces a weaker counterpart in two ways explored by the paper:

• Low-bit quantization: reducing the numerical precision of the model parameters;
• Magnitude pruning: removing parameters with relatively small magnitudes.

The original and weakened models are then contrasted during training. Their behavioral difference is used to guide learning, without requiring an external teacher or privileged information. This makes the setup particularly relevant when only unlabeled prompts are available. The on-policy aspect indicates that the training signal is connected to the model’s own reasoning behavior rather than relying entirely on fixed demonstrations prepared in advance.

Reported results

The authors evaluate WISD across multiple architectures and scales on mathematical and coding reasoning tasks. On average, the quantization variant improves accuracy over the original models by 4.0 percentage points, while pruning delivers a 5.5-point gain.

The paper reports especially large changes for some model-benchmark pairs. Pruning increases Qwen3.5-4B’s accuracy on HMMT 2025 from 57.7% to 70.5%, and on LiveCodeBench v6 from 39.1% to 56.7%. For Qwen3.6-35B-A3B, quantization raises accuracy from 68.5% to 74.6% on HMMT 2025 and from 61.8% to 69.5% on LiveCodeBench v6.

Why it matters

WISD offers a different way to think about self-distillation. The useful teacher signal does not necessarily have to come from a larger model or privileged supervision; it can emerge from a controlled degradation of the model itself. Quantization and pruning, commonly associated with compression and deployment, are therefore used here as training-time tools for producing informative contrasts.

The available material does not specify how performance changes with different bit widths, pruning ratios, or training configurations. It also does not establish that the gains will transfer uniformly to every task. Further experiments will be needed to determine how robust WISD is across models, reasoning domains, and evaluation settings.

Source: Hugging Face Daily Papers