When Does Muon Help Agentic Reinforcement Learning? Optimizer Choice Is Only Part of the Story

Lead

Muon has gained attention as a competitive optimizer for large-scale pretraining, but its role in reinforcement-learning post-training is less clear. Agentic RL is a very different setting: rewards can be sparse, trajectories are long, and progress often depends on how well a method assigns credit across many decisions. This paper, titled “When Does Muon Help Agentic Reinforcement Learning?”, asks a focused question: under what conditions does Muon actually help?

The authors study vanilla Muon in ALFWorld, a sparse-reward agentic environment, using Qwen2.5-0.5B-Instruct. Rather than presenting Muon as a universal replacement for AdamW, the paper frames the optimizer as one part of a larger training recipe that also includes the advantage estimator and learning rate.

Key takeaways

• Muon can substantially improve GiGPO results. Under Group-in-Group Policy Optimization, applying Muon only to hidden weight matrices raises final-window validation success from 0.290 with AdamW to 0.546. That is an 88% relative gain. High-learning-rate AdamW controls do not retain post-update success, suggesting that the effect is not simply reproduced by pushing AdamW harder.

• The advantage estimator matters. At a learning rate of 3e-5, Muon improves GRPO success from 0.161 to 0.268. For GraphGPO, however, the late-window gap becomes smaller when performance approaches saturation. This makes the central message more nuanced: optimizer comparisons in agentic RL cannot be separated from credit-assignment design.

• Learning rate changes the conclusion. At 1e-5, GraphGPO with Muon reaches 0.901 success, improves normalized validation AUC from 0.399 to 0.556, and reaches the 0.5 and 0.75 success thresholds 30 and 60 updates earlier, respectively. In this configuration, Muon appears to improve both final performance and learning speed.

Why it matters

The study is valuable because it brings optimizer choice back into the discussion around RL post-training for agents. Much recent work focuses on reward construction, sampling, and advantage estimation. This paper suggests that the policy optimizer can also shape stability and sample efficiency in long-horizon sparse-reward environments.

At the same time, the evidence should be read as exploratory. The comparisons are single-seed, and the experiments are centered on ALFWorld with Qwen2.5-0.5B-Instruct. The authors explicitly leave multi-seed and cross-task validation as open work. That means the result is not yet a broad claim that Muon should replace AdamW across agentic RL.

The practical takeaway is more specific: Muon may be useful when paired with the right advantage estimator and learning rate, especially in settings where long-horizon credit assignment is difficult. For teams working on RL post-training for agents, this points to a more integrated tuning strategy—treat the optimizer, estimator, and learning rate as a coupled system rather than independent knobs.

Source: Hugging Face Daily Papers