FRED Improves Future Representations for World Action Models

Introduction

A robot operating in an unfamiliar environment needs more than a reaction to the current frame. It must estimate how the scene will evolve after an action and use that prediction to choose the next control step. World Action Models, or WAMs, pursue this idea by borrowing spatiotemporal knowledge from pretrained video diffusion models and connecting future visual states with robot actions.

The challenge is efficiency. A conventional video diffusion model generally relies on multiple denoising iterations, which can be expensive for closed-loop control. A newer class of WAMs therefore performs only one denoising step, extracts an intermediate future representation, and feeds it to a separate action decoder. This is faster, but it makes policy quality highly dependent on the quality of that intermediate representation.

What FRED changes

The paper introduces FRED, short for Future REpresentation learning via self-Distillation with temporal context. Rather than attempting to generate a complete future video, FRED trains the video model to produce a useful representation of the future directly from noise.

• Ground-truth future targets: Representations extracted from actual future observations provide a concrete learning target.
• Self-distillation: The model learns to predict those future representations from noisy inputs, improving the output available after a single denoising step.
• Temporal context: Recent observations are added to the conditioning signal, giving the model information about motion, object changes, and the current stage of an interaction.
• Action-oriented integration: The resulting representation is passed to a separate action decoder, preserving the efficient WAM inference pattern.

Results and implications

The authors report that FRED consistently outperforms competing WAMs across diverse real-world manipulation scenarios. The gains are particularly pronounced on long-horizon tasks and out-of-domain environments. This suggests that better future representations may help policies maintain a more stable estimate of what will happen over several successive decisions, rather than merely improving visual prediction quality in isolation.

The strongest reported result concerns human-to-robot skill transfer. The robot tasks were absent from robot training data and were demonstrated only in human videos. FRED reached a 50.0% task success rate in this setting, compared with 19.4% for the best competing WAM. The result supports the broader premise that representations capturing future states and action consequences can provide a useful bridge between visual demonstrations and robot control.

Why it matters

FRED shifts the optimization target from generating a complete video to learning a future representation that is directly useful for action prediction. This offers a practical compromise: WAMs can retain the rich temporal priors of video models without paying the full cost of iterative video synthesis at inference time.

The available material does not specify model size, data composition, latency measurements, or detailed per-task results. Those factors will matter when assessing how well FRED transfers across robot platforms, sensors, and longer task chains. Still, the paper highlights a central design question for efficient world models: future representations must be judged not only by how plausible they look, but also by how reliably they support decisions.

Source: Hugging Face Daily Papers