RynnVLA-Latent Scales Robot Policies with 3D-Grounded Latent Actions

Introduction

Vision-language-action (VLA) models are becoming a major route toward general-purpose robot manipulation, but scaling them remains difficult. Native action demonstrations are expensive to collect, and the same behavior can be expressed through incompatible control spaces on different robot embodiments. This makes it hard to combine human videos, robot data, and simulation trajectories in a single policy-pretraining pipeline.

RynnVLA-Latent addresses the problem by inserting a shared latent-action space between visual observations and native robot commands. Instead of asking every video to provide a particular robot’s motor-level labels, the framework learns what changed in the scene and which part of that change is relevant to control.

Core approach

• 3D-grounded latent actions. RynnLAM combines residual future-feature prediction, camera-compensated 3D-flow supervision, and camera-motion decoupling. These components are intended to make the representation track controllable motion rather than superficial appearance changes.
• Video-scale supervision. The pretraining corpus contains 30.1K hours of human, robot, and simulation videos. From it, the system produces 745.52 million latent-action descriptors, without requiring native robot-control annotations for every source.
• Embodiment-specific transfer. RynnVLA-Latent uses flow matching to predict chunks of latent actions from visual observations and language. The pretrained backbone and action expert are then adapted so that the shared latent plan can be converted into the native action space of a target robot.

Reported results

The resulting 4B policy reaches 98.8% success on LIBERO and 84.7% on LIBERO-Plus without benchmark-specific adaptation. It reports 51.9% average success across five VLABench tracks and 88.6% average success on the bimanual RoboTwin 2.0 benchmark. The paper also evaluates six real-world manipulation tasks across dexterous-hand and mobile-gripper platforms, reporting the highest observed success rates among the compared methods. Frozen-representation probes and data-scaling experiments are included to test whether the learned representation is predictive of control and whether additional latent-action pretraining data helps.

Why it matters

The broader idea is more important than simply adding more video. RynnVLA-Latent treats video as a source of control-relevant transition supervision even when precise motor commands are unavailable. A 3D-grounded latent action can act as a common language connecting human behavior, simulated motion, and multiple robot bodies.

If this transfer pattern generalizes, it could reduce the dependence on costly robot-specific demonstrations and make large-scale VLA pretraining more practical. At the same time, the reported results should be interpreted in the context of the listed benchmarks and embodiment-adaptation procedures. The quality of the final conversion from a shared latent action to fine-grained native control remains an important part of the system.

Source: Hugging Face Daily Papers