SPyCE turns multimodal agent trajectories into evolving reusable skills

Introduction

Multimodal agents are moving beyond single-shot image understanding. In many tasks, they need to inspect visual evidence, manipulate images, invoke tools, and revise their reasoning over several steps. The paper “SPyCE: Skill-Policy Co-evolution for Multimodal Agents” focuses on this more demanding setting, where success depends not only on perception but also on learning reusable tool-use patterns.

The authors argue that common training approaches miss an important opportunity. Reinforcement learning methods usually compress an entire trajectory into scalar rewards, leaving the policy to rediscover useful patterns for each new task. Memory-based alternatives keep past experience, but typically use it through test-time retrieval rather than updating the policy to internalize what worked. SPyCE is designed to turn useful trajectories into structured skills that evolve with the policy during training.

Key ideas

• Trajectories become trainable assets: SPyCE distills valuable rollouts into a skill library and keeps updating that library throughout reinforcement learning.
• A hierarchical skill structure: The framework separates skills into execution skills and workflow skills. Execution skills capture local visual operations and tool-use behaviors. Workflow skills encode higher-level priors for orchestrating multi-step reasoning.
• A closed loop between policy and skills: During training, the policy conditions on retrieved skills to guide its rollouts. Better rollouts then provide material for improving the skill library, which in turn gives the policy stronger priors.
• Reported benchmark gains: Across eight benchmarks, the paper reports that SPyCE consistently outperforms both RL-based and memory-based baselines. The analysis also indicates that the hierarchical design and the co-evolution mechanism are both important.

Why it matters

The central contribution of SPyCE is reframing experience in multimodal agents. Rather than using past trajectories as one-off reward signals or static examples, the framework treats them as reusable knowledge that can shape future policy learning. This is especially relevant for agents that must repeatedly inspect images, select tools, and coordinate several operations before answering.

If the approach generalizes, it points toward a broader training paradigm: capable agents may need not only stronger policies, but also evolving skill systems that accumulate transferable procedures. For visual reasoning agents, tool-using assistants, and long-horizon multimodal workflows, joint skill-policy optimization could help reduce repeated exploration and make complex behavior more reliable.

Source: arXiv