CPO Moves RLVR Beyond Entropy With Correctness-Aware Advantage Shaping

Lead

Reinforcement learning with verifiable rewards, or RLVR, has become an important training paradigm for language models because some tasks provide clear correctness checks. A math answer, a program output, or a rule-based verifier can tell whether a final response is right or wrong. Yet a final reward is often too coarse. Training still needs a way to assign credit across the generated sequence, especially when the model must reason through multiple intermediate steps.

A common answer has been entropy-based advantage shaping. Entropy encourages uncertainty and exploration, but it has a fundamental weakness: it does not know why the model is uncertain. A high-entropy distribution may reflect healthy exploration among plausible next tokens, or it may simply mean the model is confused. The paper “Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization” proposes a different route: use distributional disagreement as a more correctness-aware signal.

Key ideas

• Entropy is not a correctness signal
  Entropy measures how spread out a probability distribution is. In RLVR, that can be useful for shaping learning, but it cannot separate productive uncertainty from detrimental confusion. This limits its value in tasks where the quality of each reasoning step matters.

• CPO compares two generation distributions
  Contrastive Policy Optimization uses token-level contrastive disagreement between a reference-guided generation distribution and a vanilla generation distribution. In practical terms, it asks how the model’s next-token preferences change when generation is guided by a reference signal compared with ordinary generation. That difference becomes the basis for correctness-aware advantage shaping.

• A finer view of token-level correctness
  Instead of relying only on final-answer rewards, CPO tries to inject correctness information into the sequence generation process. The paper provides theoretical reasoning for why the disagreement between the two distributions can indicate token-level correctness, and reports empirical evidence supporting the claim.

• On-policy distillation as a special case
  The authors also show that on-policy distillation fits inside the CPO framework. When the posterior distribution is instantiated by an external teacher model, the distillation setup becomes a particular form of CPO. This gives a unified lens for understanding policy optimization and teacher-guided learning.

• Addressing the zero-advantage problem
  RLVR can suffer when rewards are sparse or when samples fail to produce meaningful advantage differences. CPO introduces an additional distribution-level contrastive structure, which the paper argues can resolve the zero-advantage problem and make optimization more informative.

Why it matters

The broader contribution of CPO is that it reframes exploration. The question is not only whether a model should explore more, but whether the training signal can identify exploration that is likely to support correctness. If token-level disagreement reliably captures this distinction, RLVR can become less dependent on blunt entropy heuristics and more aligned with the actual learning objective.

The paper reports experiments on both in-domain and out-of-domain benchmarks, where CPO substantially outperforms entropy-based RLVR methods while maintaining strong generalization. Its analysis also suggests an interesting training dynamic: correct and incorrect responses naturally support exploration and exploitation in different ways, and balancing both leads to the best performance.

Overall, CPO offers a more targeted approach to advantage shaping for verifiable-reward reinforcement learning. It connects entropy alternatives, distillation, and token-level correctness under one framework, making it a notable contribution for researchers working on reasoning-oriented LLM training.

Source: Hugging Face Daily Papers