BPO Recasts Verifiable-Reward Training Without a Critic

Introduction

Reinforcement learning with verifiable rewards (RLVR) has become an important way to improve the reasoning behavior of large language models. In tasks such as mathematical problem solving, a model may receive a reliable signal only after it has generated a complete answer. The central challenge is therefore not merely obtaining a reward, but assigning that terminal signal to a long sequence of token decisions without making optimization unnecessarily fragile.

Bellman Policy Optimization (BPO) addresses this setting with a critic-free formulation. The method starts from Policy Mirror Descent (PMD) and uses the Bellman equations to rewrite the optimization problem as a trajectory-level objective for autoregressive generation with terminal rewards.

Key ideas

• No intermediate critic is required. Instead of estimating state values or advantages for every intermediate prefix, BPO works with the complete generated trajectory. This removes one source of modeling and estimation complexity.
• The reformulation is theoretically grounded. The paper proves that the trajectory-level objective has the same unique optimal solution as the original PMD objective. The change is therefore a reformulation of the optimization target, rather than an arbitrary replacement of it.
• The practical loss corrects policy mismatch. In actual training, the policy that produced sampled trajectories may differ from the policy currently being optimized. BPO approximates its trajectory objective and uses a smoothed ratio of complementary token probabilities for mismatch correction.
• It is evaluated in RLVR reasoning settings. Experiments on mathematical reasoning benchmarks are reported as evidence of effectiveness. The supplied material does not provide benchmark names, model sizes, or numerical gains, so it does not support claims of broad superiority over competing methods.

Why it matters

BPO offers a useful perspective on credit assignment for terminal-reward language-model training. A conventional critic must learn how valuable each partial sequence is, even when the only direct supervision arrives at the end of generation. BPO instead uses the structure captured by the Bellman equations to express the PMD objective over full trajectories. This can make the conceptual interface between terminal rewards and token-level policy updates more direct.

Its probability-ratio design is also notable. Rather than retaining the importance-sampling ratio used by Group-Relative Policy Optimization (GRPO), the practical BPO loss applies a smoothed ratio involving complementary token probabilities. The smoothing is intended to make mismatch correction usable in a long autoregressive sequence, although the supplied material does not establish how it behaves across all training regimes.

A discussion attached to the paper further notes that BPO and another method, Score Centering Stabilizes Off-policy Reinforcement Learning, share an identical core gradient while arriving at it through different derivations. Their approximations to the KL-related term differ: BPO uses a binary approximation with additive smoothing, whereas the other method uses a top-k approximation. This comparison suggests that distinct theoretical routes can produce closely related optimization signals.

Overall, BPO contributes a theoretically motivated critic-free route for RLVR. Its broader value will depend on future evaluations across tasks, model scales, and sampling conditions beyond the mathematical reasoning experiments described here.

Hugging Face Daily Papers