AffordanceWAM Turns Human Videos into a Shared Interface for Robot Manipulation

Introduction

Robot learning faces a persistent data mismatch. Robot trajectories with action labels directly supervise control, but they are expensive to collect and often limited in behavioral and environmental diversity. Egocentric human videos provide much broader coverage of everyday interactions, yet they contain no robot actions and differ from robots in embodiment and appearance. AffordanceWAM addresses this gap without attempting to translate every human motion into a robot trajectory. Instead, it uses affordances as a shared interface between the two data sources.

What the model represents

AffordanceWAM is a generative World Action Model designed to predict both how a scene will evolve and where interaction is likely to be effective. Its future-world representation contains two object-centric, spatiotemporal affordance signals:

• Scalar affordance, which summarizes the interaction relevance of an object;
• Affordance heatmaps, which identify more precise regions where an action can be applied.

These signals are generated alongside future RGB observations. They ground visual prediction in task-relevant objects and contact regions, giving the action model a more explicit basis for deciding what to manipulate and where to act.

The architecture is built on a pretrained video diffusion Transformer. World and Action Experts are parameterized separately and connected through Masked Joint Self-Attention. A unified flow-matching objective trains the system to predict future RGB observations, scalar affordance fields, affordance heatmaps, and continuous robot actions together.

How human videos contribute

Human videos supervise the three future-world streams: visual observations, scalar affordances, and heatmaps. Robot trajectories provide the same world-level supervision while adding the robot-action target. The shared learning problem is therefore not a specific human arm motion, but a more transferable description of the task: which object matters, where interaction should occur, and how the scene is expected to change.

This design removes two common requirements in human-to-robot transfer: human action labels and motion retargeting. It also lets diverse human footage improve the model’s understanding of manipulation without pretending that human and robot bodies are interchangeable.

Results and significance

Experiments on RoboCasa, CALVIN ABC→D, and real-world manipulation report consistent improvements over RGB-only and robot-data-only baselines. With robot supervision held fixed, adding more affordance-annotated human video steadily improves RoboCasa performance.

The broader implication is that human video can serve as more than generic visual pretraining. Through affordances, it can supply structured knowledge about object function, contact regions, and future scene changes, while robot data teaches how that knowledge should become executable control. AffordanceWAM therefore presents affordance as a practical bridge between video world models and embodied action generation. Its remaining challenge is to maintain reliable affordance grounding when robot embodiments, camera views, objects, and real-world conditions change.

Source: Hugging Face Daily Papers