Where Should RL Post-Training Compute Go? New Paper Breaks Down the Budget Trade-Off

Introduction

Reinforcement learning post-training is becoming a central tool for adapting foundation models to reasoning, planning, and feedback-driven robot-learning pipelines. Yet the cost of this process is often summarized by one number: total FLOPs. The arXiv paper “Where Should RL Post-Training Compute Go?” argues that this single figure can hide the most important decision: under a fixed budget, should compute buy a larger policy, more learning updates, more rollout search, or stronger reward feedback?

Key points

• Total FLOPs are not enough. The authors introduce a FLOP-accounting framework for GRPO post-training that separates compute into rollout/search, policy-update/learning, and reward- or feedback-model evaluation. This makes it possible to see whether the budget is being spent on sampling, optimization, or feedback.
• Model size changes the whole budget equation. Same-FLOP comparisons across model sizes are not straightforward. A larger policy consumes more compute per token, which means the same post-training budget can afford fewer updates or fewer rollouts. Model choice and training allocation are therefore coupled rather than independent.
• Reward systems reshape the compute profile. With rule-based rewards, most non-update compute goes into policy rollouts. With PRM-style feedback, reward-model inference becomes a visible part of the budget. Improving feedback quality therefore has an explicit compute cost that should be counted alongside policy inference and learning.
• There is no universal best allocation. Across LoRA-adapted Qwen2.5 policies, the paper finds conditional allocation frontiers. The best observed split changes with model size, total budget, reward mechanism, and evaluation target. A setting that works well in one regime may not transfer to another.
• RACE is a diagnostic tool, not a promise. The authors present RACE as a pilot-grid protocol for identifying promising allocation regimes before expensive validation runs. They explicitly frame it as a diagnostic method, not as a guarantee of held-out improvement.

Why it matters

The paper’s contribution is not a single magic ratio for RL post-training compute. Its larger message is that the community needs more transparent accounting. Two experiments may report the same total FLOPs while spending very different fractions on search, learning, and feedback. Without that breakdown, comparisons can be misleading and hard to reproduce.

For research papers, the implication is clear: report both total compute and how that compute is divided. For engineering teams, the lesson is equally practical: when budgets are constrained, run small diagnostic allocation grids before assuming that the answer is simply a bigger model, longer training, or more sampling. As RL post-training becomes more important for reasoning models, agents, and embodied systems, “where the compute went” may become as important as “how much compute was used.”

Source: arXiv