Reinforcement Learning

A new study examines how Evolution Strategies behave when used for LLM reasoning post-training. It finds that ES can improve Pass@1 while preserving broader solution coverage and achieving stronger Pass@K than GRPO.

TTPO turns majority votes into a cautious source of supervision rather than treating them as unquestionable labels. It distills agreeing reasoning traces and penalizes confident disagreements through separate optimization branches.

WarpSAC argues that off-policy reinforcement learning should not rely on one fixed set of stabilizers across training setups. It introduces two SAC variants tailored to data-limited CPU training and data-abundant GPU simulation.

Co-RL trains multiple parameter-independent models with rewards derived from their peers. The approach improves text and multimodal reasoning without ground-truth labels while reducing the collapse risks of self-rewarding RL.

Agent Lightning v1.0 presents a harnessed agentic RL paradigm in which the deployment-time agent harness directly participates in post-training. With 6K examples and modest compute, Qwen3.5-9B improved on SWE-bench Verified from 41.8% to 56.4%.

DAPD argues that a key failure mode in on-policy self-distillation comes from information asymmetry: the teacher can rely on privileged signals that the student will not have at inference time. Its dual-anchored design aims to prevent this “privilege illusion” from being distilled into the final policy.

SAF-OPD tackles a subtle failure mode in LLM post-training: reinforcement learning with verifiable rewards and on-policy distillation are complementary, but a fixed mixture can collapse entropy and suppress exploration. The proposed SAF framework regulates when and how strongly the teacher signal is used.

A new paper from Nanjing University examines token-level credit assignment in RLVR for long-chain-of-thought reasoning. Its key claim is that large probability shifts often reflect surface-form sensitivity, not true reasoning value.

Rubric-based reinforcement learning can improve open-ended LLM behavior, but its feedback can disappear in subtle ways. CriPO addresses both unexplored and suppressed criteria without relying on external guidance at inference time.

The paper addresses a core limitation of RL with verifiable rewards: it works well for math and code, but not for open-ended generation. RLSVR proposes to transform such tasks into proxy environments where rewards can be generated automatically from interaction outcomes.

Molt is an open-source training framework designed to reduce the engineering overhead of agentic reinforcement learning research. It keeps the codebase compact while supporting asynchronous training for multimodal and mixture-of-experts policies.

Skill Self-Play proposes a way to make LLM self-evolution more reliable by placing verifiable agent skills at the center of the training loop. The framework co-evolves a task proposer, a solver, and a dynamic skill controller through reinforcement learning.

The paper argues that RLVR does not substantially rewrite a model’s singular-value spectrum. Instead, the behavioral gains appear to live in the input and output singular frames, motivating a fixed-spectrum optimization stack called ISO.

Asynchronous RL improves throughput, but stale rollouts can make optimization drift away from the policy that produced the data. SAT introduces a staleness-aware trust-region rule that tightens PPO-style updates only where mismatch is most risky.

GEPO is a lightweight extension to GRPO that adjusts advantage signals using group entropy. The goal is to better handle mixed-task RL, where different prompts can require different exploration levels.

This paper rethinks post-training for open-ended tasks by upgrading judge-style feedback into coach-style experiential knowledge. Instead of compressing everything into a scalar reward, it keeps richer guidance alive in the learning loop.

Distilled Reinforcement Learning proposes a middle path between outcome-only RL and unconditional online policy distillation. It uses a teacher model to redistribute token-level policy-gradient signals while keeping optimization anchored to rewards.

SVR-R1 brings inference-time self-checking into the reinforcement learning loop. The model answers, judges its own response with a Yes/No verdict, and rethinks when it says No.

Agon proposes a competitive cross-model RL setup in which two comparable but behaviorally different models train by trying to outperform each other on the same problem.

The paper introduces Contrastive Policy Optimization, a method that replaces entropy-only advantage shaping in RLVR with a token-level signal based on distributional disagreement. The goal is to distinguish useful uncertainty from harmful confusion during model training.

A new study tests Muon in sparse-reward agentic RL and finds that its gains depend heavily on the advantage estimator and learning rate, rather than on the optimizer alone.