PLC-DPO Teaches Preference Optimization to Correct Noisy Labels

Introduction

Direct Preference Optimization, or DPO, has become a popular way to align language models because it can learn from pairwise comparisons without training a separate reward model. Its convenience, however, rests on a strong assumption: the chosen response is genuinely better than the rejected response in every training pair. Real preference data rarely satisfies that assumption. Annotators can disagree, two answers can be nearly equivalent, or the recorded preference can point in the wrong direction. Treating every label as equally trustworthy can therefore produce harmful policy updates.

The core idea

PLC-DPO, short for Posterior Label Correction DPO, reframes the problem as deciding how each supervision signal should be used. Every pair is associated with three possible latent states:

• Clean: the observed preference is credible and should be reinforced;
• Flip: the recorded direction may be wrong and should be reversed during training;
• Tie: the pair is too weak or ambiguous to justify a strong directional update.

The method derives evidence from the difference between the policy’s and the reference model’s log-probability margins for the two responses. After calibration, this policy–reference margin supplies routing weights that resemble posterior probabilities over the three states. The result is not a binary keep-or-remove decision. A suspicious example may still be useful, but its contribution can be reinforced, inverted, or suppressed depending on the inferred state.

PLC-DPO combines forward-direction, reverse-direction, and tie-regularizing losses. For stability, the reported design uses exponential moving-average calibration, a warm-up period, and confidence gating before routing decisions are applied. It reuses the log probabilities already computed by DPO and does not require an auxiliary model or additional supervision.

Results and implications

Across 57 dataset–model–benchmark cells, the paper reports a mean win rate of 60.5% against DPO, while the next-best method reaches 55.5%. Additional analyses include injected-noise tests, tie-focused stress tests, human-disagreement analysis, and self-confirmation diagnostics. According to the provided material, these experiments indicate that the routing remains stable and can distinguish flipped labels from pairs with only weak directional evidence.

The broader contribution is a change in how noisy preference data is handled. A questionable example is not necessarily useless, and filtering alone may discard information that could be recovered. By separating supervision direction from supervision strength, PLC-DPO attempts to make preference learning more fine-grained. This does not mean that the model can perfectly identify every annotation error: its decisions depend on the quality of the calibrated margin and on the stability of the policy’s own evidence. Calibration, warm-up choices, and the risk of self-reinforcing mistakes therefore remain important considerations for future evaluation.

Source: Hugging Face Daily Papers