Reinforcement Learning

DPO treats pairwise preferences as reliable supervision, but real datasets often contain reversed, weak, or ambiguous judgments. PLC-DPO uses a calibrated policy–reference margin to route each pair toward reinforcement, reversal, or tie regularization.

RadixArk has open-sourced Miles v0.1, a full-stack system designed to make large-scale post-training more verifiable, customizable, and production-ready. It connects rollouts, training, and model-weight synchronization in one architecture.

Weak-to-strong transfer does not have to mean copying a weak teacher. On-Policy Reverse Distillation uses teacher policy shifts to accelerate verifier-supported updates while preserving the student’s own optimization target.

FlowBalance combines sparse terminal verification with dense self-feedback for on-policy reasoning training. It reweights complete response trajectories so that misleading guidance on failed solutions can be reversed rather than reinforced.

Long-horizon agents often receive only one score after an entire trajectory, making it difficult to identify which actions caused success or failure. DRACO uses dynamic rubrics to redistribute trajectory-level judgments across the responsible steps for GRPO training.

A study of Countdown reasoning finds that RLVR’s loss of diversity is concentrated before the first arithmetic step. Models retain the ability to execute alternative solutions, but become less likely to enter those branches.

Astribot’s SmoothRL framework addresses a basic mismatch in robot learning: an action chunk generated by a large model is not necessarily the chunk the robot executes. It updates the policy only on actions that actually enter the physical trajectory.

A new analysis suggests that much of on-policy distillation’s benefit may come from suppressing low-probability tokens rather than transferring meaningful teacher knowledge. The authors use this finding to motivate a teacher-free method called On-Policy Self-Adaptation.

J-Zero, proposed by KAIST AI, creates a self-improvement loop among task generation, problem solving, and answer evaluation without relying on human-labeled data. The framework targets both verifiable and unverifiable tasks, with a larger reported gain in the latter.

TTPO turns majority votes into a cautious source of supervision rather than treating them as unquestionable labels. It distills agreeing reasoning traces and penalizes confident disagreements through separate optimization branches.

A new study examines how Evolution Strategies behave when used for LLM reasoning post-training. It finds that ES can improve Pass@1 while preserving broader solution coverage and achieving stronger Pass@K than GRPO.

WarpSAC argues that off-policy reinforcement learning should not rely on one fixed set of stabilizers across training setups. It introduces two SAC variants tailored to data-limited CPU training and data-abundant GPU simulation.

Co-RL trains multiple parameter-independent models with rewards derived from their peers. The approach improves text and multimodal reasoning without ground-truth labels while reducing the collapse risks of self-rewarding RL.

Agent Lightning v1.0 presents a harnessed agentic RL paradigm in which the deployment-time agent harness directly participates in post-training. With 6K examples and modest compute, Qwen3.5-9B improved on SWE-bench Verified from 41.8% to 56.4%.

DAPD argues that a key failure mode in on-policy self-distillation comes from information asymmetry: the teacher can rely on privileged signals that the student will not have at inference time. Its dual-anchored design aims to prevent this “privilege illusion” from being distilled into the final policy.

Rubric-based reinforcement learning can improve open-ended LLM behavior, but its feedback can disappear in subtle ways. CriPO addresses both unexplored and suppressed criteria without relying on external guidance at inference time.

A new paper from Nanjing University examines token-level credit assignment in RLVR for long-chain-of-thought reasoning. Its key claim is that large probability shifts often reflect surface-form sensitivity, not true reasoning value.

SAF-OPD tackles a subtle failure mode in LLM post-training: reinforcement learning with verifiable rewards and on-policy distillation are complementary, but a fixed mixture can collapse entropy and suppress exploration. The proposed SAF framework regulates when and how strongly the teacher signal is used.

The paper addresses a core limitation of RL with verifiable rewards: it works well for math and code, but not for open-ended generation. RLSVR proposes to transform such tasks into proxy environments where rewards can be generated automatically from interaction outcomes.

Skill Self-Play proposes a way to make LLM self-evolution more reliable by placing verifiable agent skills at the center of the training loop. The framework co-evolves a task proposer, a solver, and a dynamic skill controller through reinforcement learning.

Molt is an open-source training framework designed to reduce the engineering overhead of agentic reinforcement learning research. It keeps the codebase compact while supporting asynchronous training for multimodal and mixture-of-experts policies.