Not Every Token Deserves the Same Reward in Long-CoT Reinforcement Learning

Lead

Long-chain-of-thought reasoning has become a central technique for improving mathematical reasoning in large language models. Reinforcement learning with verifiable rewards, or RLVR, is one of the main training recipes behind this trend. But a basic question remains underexplored: when a full answer is judged correct or incorrect, should every generated token receive the same amount of credit?

The paper “Not All Tokens Deserve Equal Credit” argues that the answer is no. It focuses on token-level credit assignment and proposes Counterfactual Sensitivity Credit Reallocation, or CSCR, as a lightweight extension to GRPO.

Key points

• GRPO uses a coarse credit signal: Critic-free methods such as GRPO transform response-level rewards into advantages and then broadcast them across all tokens in a sampled response. This is simple and practical, but it treats a crucial algebraic step, a filler phrase, and a formatting token as if they contributed equally to the final outcome.

• Privileged self-distillation is not automatically answer-aligned: On-policy self-distillation tries to provide denser supervision by comparing an ordinary policy with a privileged self-teacher. The implicit assumption is that likelihood shifts caused by outcome information reveal which tokens should be optimized. The authors test this by holding sampled trajectories fixed and re-scoring them under two opposite outcome conditions: one asserting correctness and the other asserting incorrectness.

• Large shifts often point in the wrong kind of place: The study finds that many affected tokens move in the same direction under both outcome conditions, with limited sign reversals and substantial overlap in optimization signals. Even more importantly, the biggest shifts tend to concentrate on highly substitutable surface-form tokens, while problem-specific reasoning tokens are often less sensitive.

• CSCR reallocates, rather than replaces, credit: CSCR keeps the verifier-derived advantage direction from GRPO, but reduces the credit assigned to tokens with high counterfactual sensitivity. It then renormalizes token-level advantages so that the original credit budget is preserved. In other words, the method avoids using privileged shifts as the new reward direction; it uses them mainly to decide where not to over-assign credit.

Why it matters

The paper highlights a subtle but important failure mode in long-CoT reinforcement learning. If all tokens receive the same reward, training may reinforce stylistic or formatting habits that happen to appear in successful answers. If privileged self-distillation is trusted too strongly, the model may chase likelihood shifts that reflect linguistic substitutability rather than reasoning contribution.

CSCR is interesting because it is deliberately conservative. It does not discard verifier-based learning signals, nor does it claim to identify the exact causal importance of every token. Instead, it uses counterfactual sensitivity as a warning sign: tokens that change strongly under conflicting outcome assumptions may be unreliable places to allocate large amounts of learning credit.

According to the paper, CSCR consistently improves over GRPO and self-distillation baselines across long-CoT mathematical reasoning benchmarks with the same number of policy updates. The ablation results also support the diagnosis: privileged directions are unreliable, moderate downweighting works best, and overly strong modulation can destabilize training.

For teams training reasoning models, the takeaway is clear: better rewards are not enough. The way those rewards are distributed across the generated reasoning trace may be just as important.

Source: Hugging Face Daily Papers