DAPD Reframes Policy Distillation Around Information Alignment

Introduction

On-policy self-distillation has become an increasingly important recipe in language-model post-training. The idea is intuitive: use a stronger, privileged, or better-guided version of a policy to supervise the behavior that will later be used at inference. But the DAPD paper highlights a subtle trap in this setup. If the teacher has access to information that the deployed student will never see, the student may learn behavior that only makes sense under those richer conditions.

The authors call this failure mode privilege illusion. The student behaves as if the training-time privileged information were still present, even though its inference-time context is narrower. The result can be degraded performance rather than reliable transfer.

Key points

• The central diagnosis is information asymmetry. In conventional on-policy self-distillation, the teacher or reference policy may be strengthened by privileged signals. The student, however, must act from the inference-time context alone. DAPD treats this mismatch as the root cause of the failure.
• DAPD focuses on matched-information learning. Instead of directly transferring privilege-dependent behavior, Dual-Anchored Policy Distillation attempts to align policy behavior only when the information conditions are comparable.
• Dual-Path Anchoring builds two aligned paths. The first component, DPA, introduces a self-conditioned bridge and aligns reference and rollout behavior along two paths where the available information is matched. This is intended to block behavior that cannot be reproduced by the inference-time student.
• Dual-Source Anchoring reduces one-way dependence on the reference. DSA applies the anchoring idea in both reference-to-rollout and rollout-to-reference directions. This keeps correctness supervision while reducing overreliance on privileged reference guidance.
• Reported gains persist across scales. The paper reports that DAPD outperforms OPSD by an average of 2.00 points on Qwen3-4B across tasks. The gains are also reported at different scales, including +2.69 at 4B and +2.78 at 32B.

Why it matters

The contribution is less about a single training trick and more about a useful reframing of distillation. In language-model post-training, what the teacher knows can be as important as what the teacher outputs. If the teacher’s behavior is shaped by unavailable signals, a student trained to imitate it may inherit the wrong decision rule.

That perspective is especially relevant as LLM training pipelines combine reinforcement learning, rollouts, self-improvement, and synthetic supervision. Each added signal can strengthen the teacher, but it can also widen the gap between training-time and inference-time conditions. DAPD’s dual anchoring is an attempt to keep those conditions synchronized enough for imitation to remain valid.

The work also raises a natural follow-up question: how robust is this framework when teacher and student are not just different information views of the same policy, but genuinely different architectures or model families? A commenter on the Hugging Face discussion raised this point, and the authors noted that DAPD is currently formulated for on-policy self-distillation. They also mentioned preliminary OLMo results that appear similar to those on Qwen, with more experiments planned for a future revision.

Overall, DAPD adds an important caution to the post-training toolbox: stronger supervision is not always better if it comes from information the student cannot access. Reliable distillation may require anchoring not only the outputs, but also the information conditions under which those outputs are produced.

Source: Hugging Face Daily Papers