RynnWorld-Latent Uses Universal Latent Actions to Bridge Embodiments

Introduction

A useful robot world model must learn more than how scenes look. It must connect an action to the visual consequences that follow. This is difficult to achieve with today’s mixed datasets: robots expose different joints, end effectors, and command interfaces, while human interaction videos rarely contain labels that a robot can execute. RynnWorld-Latent tackles this problem with a cross-embodiment video model conditioned on a shared latent action representation.

The central idea

The method adopts the 3D-grounded latent actions introduced by RynnLAM as an intermediate interface. Instead of forcing every dataset into one native robot action space, it describes interaction through a representation intended to capture spatial motion and transition structure. Human videos, heterogeneous robot demonstrations, and simulation rollouts can therefore participate in joint pretraining without direct alignment of their original controls.

The model is built on a pretrained video transformer. Its conditioning design has two complementary components:

• Global action tokens provide a sequence-level description of the intended transition and its overall direction.
• A local pathway injects action information in a frame-aligned manner. It is identity-initialized, allowing the model to associate transition descriptors with the corresponding future video latents without disrupting the pretrained visual backbone at the start of adaptation.

The reported pretraining mixture contains 23 datasets and approximately 553.5 million frames. After pretraining, the model is adapted to a target robot, separating broad visual-dynamics learning from platform-specific control conditioning.

Two routes for creating data

The paper describes two ways to produce additional action-video data. In digital teleoperation, commands for the target robot are used to generate demonstrations that pair native actions with predicted or rendered visual outcomes. In latent-action synthesis, motion from a source example is transferred to an edited initial frame, and the resulting rollout is paired with retargeted pseudo robot actions. The first route is closer to the target platform’s command interface; the second can reuse motion knowledge across embodiments and environments.

On public benchmarks and two real-robot platforms, the authors report improved video-prediction metrics. Qualitative results also show action-conditioned generation when different action sequences begin from the same observation, a useful test of whether the model has learned controllable dynamics rather than merely reconstructing appearance.

Synthetic data used during intermediate training improves policy success on LIBERO and LIBERO-Plus. Both generation routes also outperform real-demonstration-only training in the evaluated real-world settings. The provided material does not include exact success-rate values, so the magnitude of these gains should be assessed from the full paper and its baselines.

Why it matters

RynnWorld-Latent moves the search for a common action space away from low-level robot commands and toward spatial, visual transition representations. This could let a world model absorb much larger and more diverse video collections before being adapted to a particular robot, potentially reducing the cost of platform-specific annotation.

There are also clear caveats. A latent action is not automatically an executable command, and pseudo actions or edited environments may differ from real physical interactions. Performance will still depend on retargeting quality, target-robot post-training, and the realism of generated rollouts. The work is therefore best viewed as a scalable interface for cross-embodiment pretraining, rather than a complete solution to general-purpose robot control.

Source: Hugging Face Daily Papers