Instead of Imitating Answers, LLMs Could Learn by Avoiding Bad Reasoning

Introduction

Improving language-model reasoning often means teaching a model to reproduce a better solution process. On-Policy Self-Distillation (OPSD) follows this idea: the model generates outputs while also using privileged information, such as a ground-truth solution, to provide a stronger teaching signal. The paper Negative Self-Distillation: Learning to Reason by Avoiding Flaws asks whether the opposite direction may be more suitable for difficult reasoning: instead of imitating an ideal trace, can a model learn by moving away from flawed ones?

Key ideas

• A critique of conventional distillation: A teacher trace conditioned on privileged information may look unusually confident and clean. If the student is forced to imitate it, the training process can suppress uncertainty, exploration, backtracking, and self-correction—the behaviors often needed for hard problems.
• A self-generated negative teacher: Negative Self-Distillation (NSD) does not depend on an external teacher or ground-truth labels. The model generates a question-specific negative condition, for example by acting as a “careless reasoner,” and the student is optimized to diverge from the resulting flawed trajectory.
• Avoiding indiscriminate unlearning: A naive unlearning objective could penalize ordinary linguistic tokens that happen to appear inside bad reasoning. That would risk weakening the model’s general language capabilities rather than improving its reasoning.
• Dynamic gating: The proposed solution uses a dynamic gate to identify reasoning-critical tokens and concentrate gradient updates on them. In principle, this separates reasoning errors from the language machinery needed to express any answer.

Why it matters—and what remains unclear

NSD reframes self-improvement as learning from failure. Its central contribution is not simply reversing the direction of a distillation loss, but treating the model’s own unsuccessful trajectories as training signals. For tasks that require search, trial and error, or revisions, avoiding a bad path may preserve more flexibility than copying a single polished solution.

The idea should nevertheless be viewed cautiously. The supplied material does not report the full algorithm, benchmarks, ablations, or quantitative gains. It therefore cannot establish whether NSD consistently outperforms positive self-distillation, nor how reliably the negative condition can be generated. A practical future design may combine both signals: use trustworthy solutions to provide direction while using negative trajectories to discourage brittle or careless reasoning patterns.

Source: Hugging Face Daily Papers