UNITAS Brings World Action Models into 3D

Introduction

A robot performing a task such as picking up an object and placing it elsewhere has to solve two coupled problems. It must determine which motion to execute, while also anticipating how that motion will change the object and its surroundings. World action models, or WAMs, are designed to model this interaction between action and physical consequence.

Many existing WAMs are built on video-generation backbones. They predict future images or visual latents after an action. That approach is useful for modeling appearance, but images are projections of a three-dimensional scene. Pixel distances do not directly correspond to physical distances, which can make them a weak interface for precise contact, grasping, and placement.

UNITAS addresses this limitation by making the interaction representation three-dimensional from the start.

Key ideas

• A shared metric frame. Within each interaction, observations, actions, and scene dynamics are represented in one metric 3D coordinate frame. This reduces the gap between visual prediction and physical motion planning.
• Actions as point trajectories. Human hands and robot grippers are represented as trajectories of 3D points, referred to as action flow. The same interface can therefore describe different robot embodiments as well as human motion.
• Scene changes conditioned on action. Scene flow describes the displacement of scene points caused by an action trajectory. The model can use this interface both to produce actions directly and to predict how the scene will change under a proposed action.
• Spatial and temporal grounding. World-aligned 3D positional embeddings attach visual tokens to locations in the interaction space, including settings without depth input. A physical-time trajectory tokenizer compresses each point trajectory into one token anchored at its current 3D position.
• Joint evaluation of prediction and control. The 1.7-billion-parameter model is reported to achieve the best action-conditioned scene prediction among compared methods on RoboTwin, with displacement errors up to 49% lower than PointWorld. The paper also reports a 99.8% success rate on LIBERO and an average result of 85% on another manipulation evaluation; the supplied material does not include the complete name of that benchmark.

Why it matters

The main contribution of UNITAS is not simply adding depth to a visual policy. It proposes a different interface between perception, action, and prediction. An action becomes a physically situated trajectory rather than a change in a sequence of images, while the predicted consequence becomes measurable motion in the scene rather than only a plausible next frame.

This design could help with two persistent challenges in robot learning. First, robot embodiments have different joints, geometries, and control spaces. A shared 3D trajectory representation offers a way to align them at the level of spatial interaction. Second, human demonstrations and robot datasets use different forms of action representation. Describing hands and grippers with a common 3D language may narrow that representation gap.

The reported results are still benchmark results and do not establish robust generalization to every open-world condition. Questions involving occlusion, sensing noise, unfamiliar objects, and long-horizon planning remain important. Even so, UNITAS presents a clear direction for embodied world models: move beyond predicting what a camera will see, and model how an action changes the physical world.

Source: Hugging Face Daily Papers