Mosaic Teaches Agents to Allocate Credit Across Sessions

Agentic systems are increasingly built around workflows that do not fit a single uninterrupted context. A model may compact an old context and continue in a fresh session, or fork a subagent to explore another line of research. These operations are useful at inference time, but they create a training mismatch when reinforcement learning still treats every rollout as one continuous session.

Why session-level credit matters

A rollout containing several sessions can be viewed as a trajectory made up of multiple training segments. A common solution is to broadcast the final outcome advantage to every segment. If the task succeeds, all sessions receive a positive signal; if it fails, they all receive a negative one. This is easy to implement, but it cannot distinguish a productive research session from a redundant one. It also provides little guidance about how many sessions should be allocated to a task.

The paper introduces Multi-Session Agentic Reinforcement Learning as a more general formulation and proposes Mosaic, a critic-free hierarchical optimization method. Its main components are:

• Trajectory-level adjustment: outcome-based advantages are modified using relative session counts within success and failure groups, bringing session usage into the optimization signal.
• Session-level shaping: the method estimates a session’s marginal information gain from changes in the log-likelihood of the ground-truth answer.
• Type-conditioned normalization: session signals are normalized by session type within each prompt group, helping comparisons across different execution roles.
• Hierarchical combination: trajectory outcomes and session-specific contribution estimates are combined into a finer-grained advantage for each training segment.

What the experiments show

The evaluation uses BrowseComp, a benchmark for challenging web-research questions, with Qwen3.5-4B in a ReAct-style agent loop. Training and evaluation cover both context compaction and subagent forking. In both settings, Mosaic performs better than the broadcasting baseline while using fewer sessions on average. Additional analysis points to improvements in session utility and allocation rather than simply increasing the number of interaction opportunities.

The broader contribution is a change in how agentic RL defines a meaningful action unit. Once a workflow can compress, restart, split, or parallelize context, the learning system must answer three related questions: Did the task succeed? Which session helped? Was the session worth its cost? Mosaic offers a practical framework for connecting these questions without adding a separate critic. It also suggests that session scheduling and credit assignment should be optimized together. The evidence is currently centered on one web-research benchmark, model family, and execution setup, so broader tests will be important for assessing how robust the approach is across tasks and orchestration designs.

Source: Hugging Face Daily Papers