OneWM Tests Whether World Models and Action Learning Can Improve Each Other

Introduction

World models and policies are often developed as separate components in embodied AI. A world model predicts how an environment may change, while a policy selects the next action. OneWM explores whether these capabilities can be trained together rather than optimized in isolation. Its central idea is a shared video-action architecture with a unified causal training objective.

How the framework works

Within OneWM, an action plays two roles. It can be supplied as a condition for predicting the visual consequences of an intervention, and it can also be learned as the output required by a policy. The model therefore has to connect two questions: “What may happen if this action is executed?” and “What action should be generated from the current observation?”

This design does not simply add a video-generation module to a controller. It is intended to test whether learning forward dynamics helps action learning, and whether learning actions makes the learned dynamics more useful. The study evaluates both policy learning and world modeling on LIBERO, LIBERO-Plus, and RoboCasa365, followed by experiments with physical robots.

Main findings

• Action-conditioned prediction helps policies. Learning the visual consequences of specified actions improves policy performance and generalization when scenes change.
• Policy learning helps world models. Learning to generate actions improves the fidelity of multi-view future prediction under the same architecture.
• The benefit is reciprocal. The results support a training loop in which action understanding strengthens prediction, while prediction provides a more useful representation for control.
• The idea is tested beyond benchmark videos. The project includes real-world demonstrations such as stacking colored blocks, placing an object into storage, and preparing a simple breakfast.

Why it matters

The work offers a practical argument for combining world modeling and policy learning. A controller that only matches visual patterns may struggle when the scene changes. By learning what different actions are expected to do, it can acquire a more causal relationship between observations, interventions, and future states. Conversely, action prediction may force the model to preserve the parts of a scene that matter for control, rather than optimizing visual plausibility alone.

There are important limits to the available evidence. The supplied material does not report exact improvement numbers, nor does it establish how the method behaves over very long rollouts, across substantially different robot embodiments, or under difficult contact dynamics. The project’s “Endless Dream” demonstrations are model-generated rollouts from initial observations; predicted controls are not executed in a simulator and should not be interpreted as physical control trajectories. The evidence therefore supports the promise of unified causal training, not a claim that long-horizon planning has been solved.

Overall, OneWM makes a focused contribution: it frames action learning and world modeling as mutually reinforcing objectives within one architecture. Future work will need stronger uncertainty estimates, longer-horizon evaluation, and more interactive real-world testing to determine how far this reciprocal advantage extends.

Source: Hugging Face Daily Papers