SEED Turns Agent Trajectories Into Self-Evolving Hindsight Skills

Introduction

As large language models are increasingly trained as interactive agents, they must do more than produce a single answer. They need to operate across multiple turns, use tools, interpret environment feedback, and make decisions whose consequences may only become visible much later. Outcome-based reinforcement learning is a practical way to optimize such systems, but it has a weakness: rewards are often sparse and assigned at the trajectory level.

That creates a supervision gap. A successful episode tells the model that the overall behavior worked, but not which intermediate observation, tool call, or reasoning step mattered most. SEED, short for SElf-Evolving On-Policy Distillation, proposes a way to turn completed trajectories into training-time guidance.

Key Ideas

• Learning from completed trajectories: SEED first fine-tunes the policy to inspect full interaction traces and generate natural-language skills. These may describe reusable workflows, decisive observations, or rules for avoiding failure.
• Self-evolving analysis: During reinforcement learning, the same current policy gathers trajectories and acts as the analyzer that extracts hindsight skills from them. As the policy improves, its skill extraction ability can improve as well.
• Dense token-level supervision: SEED re-scores sampled actions under two settings: the ordinary context and a skill-augmented context. The probability shift caused by adding the skill is converted into a dense on-policy distillation signal.
• Joint optimization with RL: The auxiliary distillation objective is trained alongside outcome-based reinforcement learning, rather than replacing it. This keeps the additional supervision aligned with the policy’s current trajectory distribution.

Why It Matters

The interesting part of SEED is that it treats hindsight as a renewable source of supervision. Instead of relying on a fixed teacher or handcrafted annotations, the agent learns to summarize its own experience into portable skills and then uses those skills to shape future behavior.

This is especially relevant for long-horizon agentic tasks, where feedback is delayed, exploration is expensive, and many failures arise from small intermediate mistakes. If a model can extract useful lessons from its own rollouts, it may learn not only whether an episode succeeded, but also which decision patterns are worth repeating or avoiding.

According to the paper summary, SEED improves performance and sample efficiency across text-based and vision-based agentic tasks, while showing robust generalization to unseen scenarios. The next question is how well this self-generated skill supervision holds up in more open-ended environments, longer tool chains, and noisier real-world feedback loops.

Source: Hugging Face Daily Papers