RoboTryOn Turns Human Videos into Robot Demonstrations

Robot foundation models have made robot control more capable, but adapting them to a new task still commonly requires teleoperated demonstrations. This creates a practical bottleneck: collecting high-quality demonstrations is expensive, slow, and difficult to scale across robot platforms. RoboTryOn explores whether the much larger supply of human activity videos can serve as an alternative source of robot training data.

Treating embodiment transfer as virtual try-on

The paper’s central idea is captured by its name. In virtual try-on, the person and scene remain while the garment changes. RoboTryOn applies the same intuition to human-to-robot translation: it keeps the demonstrated task, manipulated objects, background, and motion intent, but replaces the human performer with a target robot specified by a reference image.

The input is an egocentric human demonstration. The output is a synthesized video that depicts the target robot carrying out the same activity, together with action information intended for policy learning. This differs from simply overlaying a rendered robot on top of a human recording. It also goes beyond directly retargeting human poses into robot joint commands, an approach that can leave substantial visual and control gaps for high-degree-of-freedom embodiments.

Keyframes and hand motion guide synthesis

RoboTryOn first translates selected keyframes in an autoregressive manner. These translated frames then guide video synthesis, helping preserve the robot’s appearance and maintain visual consistency across time. The generation process is additionally conditioned on the human hand skeleton, so the synthesized robot follows the important hand movements that define many manipulation tasks.

Generating a plausible video is only part of the problem. A policy also needs action labels. The authors therefore use an inverse dynamics model to infer actions from the synthesized demonstrations. The model incorporates depth features, which can provide spatial cues that are ambiguous in ordinary RGB frames, such as relative distance and the geometry of interactions with objects. This turns the output from a visual imitation into a more complete synthetic training example.

Reported results and practical significance

Experiments across multiple humanoid embodiments include novel tasks and challenging bimanual manipulation. According to the paper, policies trained with RoboTryOn data improved average success from 2% to 60% without real robot demonstrations for those tasks. The reported performance also matched policies trained solely on teleoperated demonstrations, while requiring approximately one-third of the data collection cost.

The broader contribution is a possible bridge between human video at internet scale and robot data that can support control learning. If effective, the approach could reduce the need to teleoperate every robot for every task and make cross-embodiment data generation more accessible.

There are important limitations to keep in view. A visually consistent synthesized demonstration does not automatically guarantee physically correct contact dynamics, force behavior, or executable joint trajectories on a real robot. Occlusion, object interaction, embodiment differences, and errors in inferred action labels may all affect deployment. Future validation will need to establish how reliably the generated data transfers to real-world execution and to robot forms beyond the experimental settings reported here.

Source: Hugging Face Daily Papers