Are Dense Rewards Worth the Extra Compute in GRPO?

Reinforcement learning with verifiable rewards trains language models by rewarding a correct final answer. In standard GRPO, that outcome-level signal is effectively shared across the tokens in a solution. Dense process rewards aim to improve credit assignment by evaluating intermediate steps. One example is a Monte Carlo reward: stop at a step, sample continuations from that point, and use their outcomes to estimate how promising the step is.

The idea is attractive, but the estimate has a substantial price. In the study's setup, sampled continuations account for 91% of all generated tokens. A comparison made per parameter update therefore gives the dense method more computation than GRPO. Likewise, a gradient test on a single batch can reveal which intervention changes the update most, while saying little about whether that intervention is efficient over a full training run.

A compute-matched evaluation

The authors introduce a protocol that matches interventions by both generated-token budget and training GPU-hours. The experiments were preregistered and included controls for side effects. Across 1,668 runs on six models, matching compute reversed conclusions suggested by per-update comparisons.

On Qwen2.5-1.5B, the Monte Carlo reward looked better than standard GRPO when measured after the same number of updates. Once the generated-token budget was matched, however, its final accuracy was 14.3 percentage points lower. Under equal training time, it was 7.8 points lower. None of four dense-reward variants outperformed standard GRPO even when each received the same amount of training time.

A per-token gradient test was not useless: it ranked the reward variants in the same broad order as full training. But it overestimated their eventual benefits by one to two points. That error is comparable to the improvements such a test is intended to detect, making it a poor substitute for compute-matched end-to-end evaluation.

Where a simpler intervention helped

The study also examined data selection rather than a new reward. It prioritized prompts for which the model's predicted probability of success was close to one-half. These are cases where the model is uncertain and may have the most room to learn. At matched compute, accuracy increased by 1.7 points on Qwen3-0.6B and 0.8 points on Qwen2.5-1.5B. The gains replicated with fresh seeds and unseen problems.

The effect was not necessarily evidence of a fundamentally superior training algorithm. The improvements appeared early, and standard GRPO with twice the initial learning rate came within half a point. A simple rule—that candidate prompts were predicted to fail more often than succeed—also predicted where the intervention would appear in five of six preregistered tests.

Why it matters

The results do not establish that dense rewards are always ineffective. They show that their value must be judged against the computation required to produce them. For reasoning models, richer credit assignment can be outweighed by the rollouts needed to estimate it. Future comparisons of reward functions, data selection methods, and training interventions should therefore report matched generated tokens and GPU time, rather than treating an update, batch, or gradient as the unit of progress.

Source: Hugging Face Daily Papers