Reinforcement Learning

A new study finds that additive rubric scoring can reward models for piling on irrelevant advice while missing the decision that matters most. ProRubric changes the aggregation protocol so related requirements must succeed together.

TIPS trains generative process reward models using only whether the final judgment is correct. With 3.2K outcome-labeled trajectories, its Qwen3-based model reaches 85.2 F1 on ProcessBench.

KlingAI Research introduces DiffusionART, an online reinforcement learning method that aggregates reward-improving directions across samples for each prompt and uses drift-adaptive restoration. The paper reports substantially lower GPU costs while matching or exceeding competing methods in image and video generation.

Xiaomi has detailed the post-training system behind MiMo-V2.6, where a single reinforcement-learning step can generate billions of tokens and cost millions of dollars. The report highlights that scaling Agentic RL requires better graders, safer environments, broader harness coverage, and more resilient infrastructure.

The MiMo-V2.6 technical report presents an infrastructure-oriented approach to scaling reinforcement learning for multimodal foundation models, spanning throughput, environments, and grading.

Self-Retrospection Distillation (SRD) uses completed trajectories to supervise what an agent should have anticipated before acting. It is designed to recover learning signals when group-relative reinforcement learning sees identical rewards across all rollouts.

Self-generated practice questions can help reasoning models improve, but invalid tasks and hidden mathematical duplicates may eventually cause performance collapse. R-Quest adds validity and novelty feedback to keep the learning loop productive.

TRACE addresses the mismatch between training and rollout quantization in reinforcement learning for MoE language models. It uses rollout-side quantization outcomes to guide training-side FP4 rounding and reports up to 5.4× rollout speedup while preserving performance close to BF16 rollouts.

A study of cross-tokenizer on-policy distillation finds that broader alignment does not automatically improve learning. Compact supervision at strictly aligned positions can outperform attempts to supervise every mismatch span.

NeMo-DCR synchronizes only changed parameter values between training and rollout clusters while preserving bit-exact equivalence with a dense checkpoint load. In a 1-trillion-parameter test with a 3% change rate, refitting fell from 87.5 minutes to 2.5 minutes.

HuatuoGPT-3 introduces OnePO, which uses teacher outputs as temporary guidance rather than a permanent imitation target for direct domain adaptation from base models. The paper reports a score of 71.4 for its 27B model on HealthBench Professional.

LoGRA compresses useful reinforcement-learning gradients into low-rank sketches and combines them with predicted-KL step control. The paper reports up to 45.7% lower average training memory while maintaining performance on reasoning tasks.

A study of flow-based VLA models finds that reinforcement learning changes are strongly concentrated in a small action-expert component called Timestep Modules. Its shift-vector updates also encode task success and transfer relationships.

A systematic study maps how on-policy distillation transfers reasoning across teacher and student scales. It finds that compact RL experts can improve larger students, while teacher size and score alone do not determine supervision value.

RLVR training can waste rollout capacity when every sampled answer in a group fails. GRAFT addresses this gap by exchanging complementary trajectories between heterogeneous models while explicitly controlling off-policy mismatch.

A new study examines why training and inference engines can assign different probabilities to the same token during RLVR. Its Calibrated Importance Sampling method uses confidence-aware clipping to control the resulting policy-update error.

A Tsinghua NLP team introduces Diffusion Reward Model, or DRM, which treats reward modeling as conditional density estimation rather than single-value prediction.

Qwen introduces QwenGyre, an end-to-end framework for online reinforcement learning on extreme-long-horizon agent tasks. It combines elastic GPU allocation with branch-aware trajectory processing and deduplication.

VLA-Precision addresses policy drift and system overhead in real-world online reinforcement learning for vision-language-action models. Its ACoB algorithm combines staged value calibration with reference-regularized updates, while ACoB-Stream targets the throughput bottleneck of large VLAs.

Tool-calling agents mix structured API actions with natural-language summaries, but standard GRPO often applies one trajectory-level advantage to both. SLCA-GRPO separates these signals by output segment to reduce harmful interference.

Rufus-Air presents an open, sequential post-training recipe built on GLM-4.5-Air-Base. Its contribution is not only a model result, but a detailed view of how data, rewards, stage order, and infrastructure work together.