CoPE Trains Language Agents to Act and Forecast Together

Introduction

Language agents often need to complete tasks through long sequences of observations, tool calls, and decisions. In standard agentic reinforcement learning, the main learning signal may arrive only after the task ends as a single scalar reward. That signal indicates whether the trajectory succeeded, but it provides limited guidance about which intermediate decisions helped the agent get there. CoPE, short for Co-Evolving Acting and Forecasting for Language Agents, proposes a way to reuse more of the information already present in successful trajectories.

How CoPE works

Rather than relying only on a terminal reward or adding a separate value estimator, CoPE introduces multi-step action targets. On successful on-policy trajectories, the policy backbone is trained to predict the actions that actually follow a given history. The same trajectory can therefore supervise the model from multiple points in time:

• More direct intermediate supervision: later actions in a successful trajectory become training targets instead of being used only indirectly through the final reward.
• A shared acting-and-forecasting model: the backbone learns both to choose the next action and to anticipate future actions from preceding histories.
• No extra inference procedure: at deployment, the agent still selects one action at a time. Forecasting is a training objective, not an additional runtime module.

This differs from two common ways of exploiting trajectory information. Per-step value targets still generally originate from the same terminal signal, while observation targets—such as web pages or terminal logs—can be long, noisy, and limited to the next environment response. CoPE instead predicts action sequences, keeping the auxiliary objective closer to the agent’s actual decision process.

Results and implications

The paper evaluates CoPE on ALFWorld, ScienceWorld, WebShop, τ²-bench, and AppWorld. Against GRPO and reinforcement-learning baselines with predictive auxiliary targets, CoPE reports stronger aggregate performance in simulated environments and improved overall results on the realistic application benchmarks. Its sample efficiency reaches up to approximately twice that of some comparison methods.

The broader significance is that improving long-horizon agent training may not require simply collecting more rollouts. A successful trajectory already contains many potential learning targets if the training objective is structured to expose them. By repeatedly asking the policy to infer what should happen after each history, CoPE turns future actions into an additional source of credit assignment and allows acting and forecasting to co-evolve within one model.

There are also open questions. The approach is built around successful on-policy trajectories, so the quality and diversity of those trajectories may affect what the model learns. The supplied material does not include detailed ablations or failure analyses, meaning the method’s limits across different task distributions still require further study.

Hugging Face Daily Papers