HuRo Turns Human Videos into Scalable VLA Pretraining Data

Introduction

Vision-language-action models need more than images and instructions to control robots reliably. They also require action trajectories that connect what the robot sees with what it should do. Collecting such data on real robots is expensive and usually covers a limited range of tasks, objects, and environments. Human videos offer a much broader source of interaction data, but they are not directly interchangeable with robot demonstrations: the camera viewpoint, body structure, and motion capabilities are different.

HuRo investigates whether human videos can become useful supervision for VLA pretraining after being transformed into a robot-oriented representation. The paper’s central idea is that visual alignment and action alignment should be handled together rather than treated as independent problems.

The robotization pipeline

HuRo processes heterogeneous human videos through a pipeline designed to produce robot-aligned observations and trajectories. Its main components are:

• Visual robotization: Human-video observations are converted into signals that better match a robot’s viewpoint and perception setup, reducing the gap between human egocentric footage and robot camera observations.
• Motion retargeting: Human manipulation motions are transformed into action trajectories that can serve as robot-oriented supervision instead of using the video only as visual data.
• Intermediate-signal inference: Because different video sources provide different levels of annotation, the pipeline infers missing intermediate information across annotation levels.

Using this process, the authors build HuRo from five human-video sources. The dataset contains about 630,000 robotized episodes and 142 million processed frames, providing a large-scale test bed for studying whether robotized human experience can support VLA pretraining.

Results

The evaluation covers four real-world manipulation tasks. As the amount of robotized pretraining data increases, overall completion rises from 51.5% to 80.3%. Under out-of-distribution conditions involving spatial and visual shifts, completion improves from 34.9% to 72.2%. These results suggest that the benefit is not limited to memorizing familiar scenes; the additional data can also improve adaptation to changes in location and appearance.

The ablations clarify how the gains are obtained. Visual robotization improves robustness under distribution shifts, indicating that the observation format matters. At the same time, end-to-end pretraining with retargeted actions outperforms visual-only transfer. In other words, transferring what the scene looks like is helpful, but transferring how an interaction unfolds is more valuable when the goal is robot control.

Why it matters

HuRo points to a scalable alternative to collecting every training example with a physical robot. Human videos can provide broad information about object interactions, task structure, and visual variation, while robotization makes that information more compatible with a target embodiment. This could allow real-robot data to be reserved for calibration, execution-specific refinement, and evaluation.

The approach does not remove all challenges. Robotization depends on inferred information, so the quality of the generated observations and trajectories remains important. Different robot embodiments may also require further adaptation, and human demonstrations do not automatically guarantee executable robot actions. HuRo is therefore best understood as a complement to real-robot data rather than a replacement for it. Its main contribution is showing that, when observation and action are aligned together, human videos can become meaningful robot experience for large-scale VLA pretraining.

Source: Hugging Face Daily Papers