ACCRUE Helps Robots Learn New Skills Without Forgetting Old Ones

Introduction

Robot foundation models can provide broad manipulation capabilities from demonstrations, but deployment rarely ends after the first successful policy. A robot may need to learn a sequence of new tasks while continuing to perform skills acquired earlier. This creates a difficult trade-off: adaptation should be sample-efficient, yet updates must not overwrite useful behavior.

Existing continual reinforcement learning approaches for vision-language-action systems often address forgetting through on-policy updates. Off-policy reinforcement learning, meanwhile, can make better use of collected transitions for an individual task, but does not necessarily preserve a learner across a sequence of tasks. ACCRUE, short for Continual Off-Policy Reinforcement Learning for Robot Foundation Models, combines these two goals.

How ACCRUE works

• A frozen foundation model. During reinforcement learning, the large robot foundation model is kept fixed. Only a compact actor-critic module is trained to refine the actions produced by the base policy. This limits additional GPU training and avoids repeatedly changing the model’s visual and behavioral representation.
• An inherited learner. After one task is completed, the actor and critic are carried into the next task instead of being reinitialized. The inherited critic is adapted using transitions from the current task and supplies value guidance for actor learning. The actor brings forward behavior already acquired through reinforcement learning.
• A small memory for behavioral retention. The system keeps a limited number of episodes from earlier tasks. During learning on a new task, the actor is distilled toward cached behavior targets from those episodes. The goal is not to replay the entire history, but to provide a compact constraint against severe drift.
• Compatible replay. ACCRUE uses fixed visual features so that the inherited learner and its replay data remain compatible across base-policy updates. This helps make off-policy continual learning more stable.

Results on simulated and real robots

The study evaluates ACCRUE on contact-rich assembly sequences in simulation and on physical robot hardware. In the real-world setup, each task begins with a small set of demonstrations, followed by reinforcement learning interaction collected directly from the robot. Using π0.5 with FastTD3, later tasks reached the stopping criterion of 95% success over the latest 20 episodes after roughly 4 to 12 minutes of robot interaction.

Compared with running single-task reinforcement learning independently for each task, ACCRUE reduced robot interaction by 31.0% to 68.5% on the AutoMate assembly suite. On the NIST assembly suite, the reduction was 38.7% to 42.9%. The reported interaction time includes online warmup but excludes demonstration collection and foundation-model fine-tuning, so it specifically measures the robot cost of the online RL stage.

Retention was also a central result. After a complete task sequence, the final policies achieved 118 successes out of 120 trials across six earlier tasks, or 98.3% overall. The authors additionally evaluated ACCRUE in simulation with GR00T and FastSAC, indicating that the framework is not tied to a single foundation-model and optimizer pairing.

Why it matters

ACCRUE reframes robot post-training as an accumulating process rather than a series of isolated adaptations. Its main practical choice is to preserve large-model generality while allowing a small, portable RL learner to store task-specific improvements. That is attractive for real robots, where physical interaction is slower, more expensive, and more constrained than offline computation.

The results should still be interpreted within their scope. The experiments focus on assembly sequences and particular combinations of foundation models and RL algorithms. Transfer may depend on the similarity between tasks, the size of the retained replay, and the stability of fixed visual features. Broader task sequences and longer-term deployment will be needed to test how well the approach scales.

Even with those caveats, ACCRUE offers a clear design direction for persistent robot learning: inherit value and behavior knowledge, use a small memory to protect earlier capabilities, and spend new robot interaction primarily on the skills that still need improvement.

Source: Hugging Face Daily Papers