CriPO Uses Self-Distillation to Recover Lost Signals in Rubric-Based RL

Introduction

Rubric-based RL has become an appealing direction for improving large language models on open-ended tasks. Instead of compressing quality into a single vague score, it breaks evaluation into multiple criteria, such as correctness, completeness, or domain-specific requirements. In principle, this should provide richer supervision for post-training.

The paper argues, however, that richer rubrics do not automatically produce richer optimization signals. Some criteria may never be satisfied by any rollout, leaving the policy with no signal to learn from. More subtly, some criteria may be satisfied by a subset of rollouts but still fail to influence learning because the final aggregated reward assigns those rollouts non-positive advantages. The authors call these two cases Unexplored Criteria and Suppressed Criteria, and introduce Criterion-Distilled Policy Optimization, or CriPO, to address both.

Key points

• Unexplored Criteria receive no feedback. If no sampled response satisfies a rubric item, standard RL has no positive example from which to reinforce that behavior.
• Guided rollouts can create mismatch. Prior exploration-oriented methods may inject rubric information during rollout generation. This can help exploration, but it trains the model on externally guided trajectories while inference runs without that guidance, creating a train-inference gap in autoregressive decoding.
• Suppressed Criteria are a separate failure mode. A rollout can contain behavior that satisfies a criterion and still be assigned a non-positive aggregate advantage. In that case, useful criterion-specific tokens may be weakened rather than reinforced. The paper reports that more than 57% of samples show this issue during training, with an average of 1.8 suppressed criteria per sample.
• CriPO distills criterion-specific behavior back into the policy. For unexplored criteria, it builds a criterion-injection self-teacher and applies a localized forward-KL loss. For suppressed criteria, it uses a counterfactual self-teacher to identify relevant tokens in negative-advantage rollouts and flips their token-level advantages to preserve useful patterns.

Why it matters

CriPO is notable because it tries to internalize rubric guidance rather than depend on an external helper at inference time. This matters for LLMs: once generation begins, small early deviations can compound across subsequent tokens.

According to the paper, CriPO consistently outperforms existing rubric-based RL methods on medicine and science benchmarks, while reaching comparable performance with roughly two times fewer optimization steps. The result suggests that improving how criterion-level signals flow through training may be as important as designing better rubrics themselves.

More broadly, the work highlights a limitation of scalar reward aggregation. A multi-criteria rubric does not guarantee multi-criteria learning unless the optimization process can preserve local evidence of each criterion. CriPO’s contribution is to make those local signals more visible to the policy.

Source: Hugging Face Daily Papers