Zero-WAM Lets Robots Generalize to New Tasks by Watching Humans

Introduction

A robot that performs familiar skills reliably is not necessarily a robot that can work in an open-ended environment. The harder problem is cross-task generalization: executing a manipulation task that was not present during training. Large language models offer a useful analogy. They can often adapt to a new request by reading it in context, without changing their parameters. Zero-WAM asks whether a similar interface can be built for robot manipulation, with a human video serving as the task description.

What Zero-WAM does

Zero-WAM is a causal video-action model. At execution time, it receives a human demonstration as an in-context prompt and predicts the robot’s future actions. The video is not treated merely as a trajectory to imitate. It is intended to convey how the task should evolve over time: which objects matter, what sequence of interactions is required, and how the manipulation progresses. This lets the model use the prompt to specify a task that may not have appeared in its training set.

The approach addresses a major data bottleneck. Large collections of robot trajectories do not automatically provide corresponding human demonstrations, while collecting carefully aligned human-robot pairs is expensive. The authors therefore propose an automatic pipeline that converts task-sampled robot trajectories into semantically matched human videos. The resulting HumanGen dataset contains 74.2K human-robot in-context learning pairs covering 8.6K tasks.

Training against shortcuts

Simply adding a video prompt does not guarantee that a policy will use it. A model can instead exploit correlations from tasks seen during training and ignore the contextual demonstration. To counter this behavior, Zero-WAM introduces an in-context future chunk prediction, or IFP, objective. The training signal is designed to suppress shortcuts associated with known tasks and encourage the policy to draw task information from the video before predicting future action chunks.

Results and implications

On seven unseen tasks in the RoboTwin 2.0 simulation, Zero-WAM achieves a 47.0% average success rate. This is an absolute improvement of 29.5 percentage points over the strongest video-action baseline reported in the supplied material. The paper also describes real-world evaluations in which the robot follows human video guidance for unseen tasks, but the available excerpt does not include the detailed task breakdown or success rates, so stronger claims about real-world performance would be premature.

The broader contribution is a shift in how robotic task specification is framed. Language is compact and symbolic, but a video can show geometry, timing, contact, and intermediate states without requiring every detail to be verbalized. HumanGen and IFP target two complementary obstacles: the lack of task-rich paired data and the tendency of policies to memorize training-task shortcuts.

There are still open questions. Human demonstrations and robot execution differ in embodiment, camera viewpoint, control space, and physical capabilities. Performance may also become more difficult as tasks involve longer horizons or less predictable environments. Even so, Zero-WAM suggests that “watch and act” can be a practical route toward parameter-free task adaptation in embodied systems.

Source: Hugging Face Daily Papers