Turning Memory into Plans: A New Route to Long-Horizon Robot Control

Introduction

A robot performing a long manipulation task needs more than the current camera frame. It may need to know which subtasks have already been completed, where an object appeared earlier, and whether the current stage has actually finished. These dependencies make many real-world tasks non-Markovian, while conventional policies often remain centered on the present observation.

MaP-WAM, short for Memory as Plans, proposes a different role for memory. Instead of repeatedly conditioning the action policy on the entire past, it first turns long-term experience into an executable plan and then lets a policy act on that plan.

Core ideas

• Episodic segment records: Completed task segments are stored with language instructions and sparse visual context. This preserves more visual evidence than a purely linguistic summary while avoiding an ever-growing visual window.
• Planning separated from execution: The framework divides memory-dependent world-action modeling into memory-grounded planning and plan-conditioned execution. The planner uses long-term multimodal context to produce the next segment-level language plan and corresponding visual guidance.
• Actions and progress predicted together: A World-Action-Progress model predicts action chunks as well as the progress of the current plan. Since different segments may take different amounts of time, transitions are not tied to a fixed duration.
• Closed-loop transitions: Plan-observation alignment is used to calibrate predicted progress. When current observations indicate that a segment is complete, the system can update its context and request the next plan.
• Bounded execution context: The executor keeps a fixed context length. Structured attention also supports key-value caching during both planning and execution, reducing the need to repeatedly process the full history.

Why it matters

The central contribution is a change in how robotic memory is represented. Memory does not have to remain as a raw sequence attached to every action decision. It can become an intermediate plan that combines language-level intent with selected visual evidence. This design seeks to preserve information needed for cross-stage decisions while controlling the cost of long-context execution.

The approach also exposes important open questions. A planner may omit a visual detail that later proves essential, and unreliable progress estimation could cause a segment transition too early or too late. The quality of plan-observation alignment is therefore as important as the quality of the generated plan. MaP-WAM should be viewed as a system design direction, not a complete solution to long-term memory.

The paper reports state-of-the-art performance on RMBench. Beyond that result, its broader message is that robotic policies may benefit from transforming history into actionable structure rather than attempting to preserve every past observation at execution time. This could be particularly relevant to long-horizon, multi-stage manipulation.

Source: Hugging Face Daily Papers