MPO Aligns Long-Horizon Agent Training with the Turn Structure

Training a long-horizon language-model agent requires more than rewarding the final answer. A trajectory may contain many assistant turns, tool calls, observations, and recovery attempts. The practical challenge is to determine which action changed the outcome and how responsibility should be assigned inside that action.

Many policy-optimization methods operate at the token level. That granularity is useful, but it can be poorly matched to agent interaction. A complete assistant turn may include a rationale, a structured argument, and a tool invocation. If the invocation contains a malformed parameter, spreading the same training signal across every token in the turn can punish text that was otherwise correct. It may also make long trajectories noisier when turns differ greatly in length.

The paper introduces Multiscale Policy Optimization, or MPO, to address this mismatch. Its starting observation is that critic values tend to vary mainly between assistant turns, while remaining comparatively stable within a turn. MPO therefore anchors optimization at the turn level, while retaining finer token-level distinctions where they are useful.

The method has two main components:

• Multiscale Advantage Estimation (MAE), which represents the effect of an action at the turn scale and refines credit assignment among the tokens that compose it.
• Turn-level importance sampling, which aligns policy-ratio calculations with turn-level updates rather than treating every token as an independent optimization unit.

The key idea is not to replace token-level learning with a coarse turn label. Instead, MPO separates two related questions: which complete action deserves credit or blame, and which parts of that action contributed to the result. This structure is especially relevant to search, tool-use, and automation agents, where an assistant turn often corresponds to a semantically meaningful operation.

According to the supplied material, MPO delivers consistent improvements over strong reinforcement-learning baselines across those task families, using both 4B and 30B-A3B models. The available discussion also suggests that turn-level credit alone can be useful and that token-level refinement provides an additional benefit. These claims should still be interpreted with care: broader multi-seed studies, varied turn lengths, and checkpoint-selection protocols would be important for assessing robustness.

MPO’s broader significance is a shift in the unit of optimization. Agent training may benefit from respecting the same action boundaries that appear in deployment. At the same time, the approach raises practical questions about importance-ratio clipping for unusually long turns and about whether critic values remain calibrated as the policy changes. If those issues are resolved, turn-level values could potentially support failure detection, retries, or human review at inference time—but that application requires evidence beyond the current summary.

Source: Hugging Face Daily Papers