Teaching Robot Hands to Operate Like Humans: Morphometric Imitation Bridges Retargeting and Zero-Shot Deployment

Introduction

Human hand-object interactions contain rich information about grasping, turning, pressing, and coordinated finger movement. That makes them an attractive source of demonstrations for dexterous robots. Yet transferring these demonstrations directly to a robot hand is difficult. Human and robot hands differ in link lengths, joint layouts, ranges of motion, and physical capabilities. A trajectory that looks visually similar may also lose crucial contacts with the object or violate the robot’s dynamical constraints.

Researchers from UC Berkeley and collaborating institutions propose Morphometric Imitation to address these issues as a staged pipeline. The approach separates morphological adaptation from dynamic correction and policy learning, rather than treating human motion retargeting as a single geometric mapping problem.

How the framework works

• Morphology-aware retargeting. The first stage, called morphometric optimization (MMO), maps human motion to different robot-hand structures. Its objective is not merely to match joint trajectories. It also preserves the contact relationships demonstrated between fingers and the object, reducing failures in which the robot appears to copy the pose but no longer maintains the intended grasp.
• Dynamic correction with residual RL. Kinematic retargeting alone does not guarantee that a robot can execute the motion. The second stage applies residual reinforcement learning on top of the kinematic reference. It uses object pose and contact information from the human demonstration to refine the motion and generate robot demonstrations that are more dynamically feasible.
• Distillation into visuomotor control. In the final stage, the corrected demonstrations are distilled into visuomotor policies. The deployed policy can therefore produce actions from visual input, instead of requiring the complete human-motion reconstruction and retargeting pipeline at run time.

Reported results

The study evaluates the method with three robot hands and ten human hand-object interaction tasks. Against the strongest of five baselines, MMO improves contact F1 by at least 8 percentage points for every tested hand. It also increases the success rate of downstream dynamic retargeting by as much as 35 percentage points. Ablation results indicate that object pose and contact information provide complementary benefits for residual reinforcement learning.

The learned visuomotor policies are evaluated in the real world through 300 trials involving 30 objects. They achieve an 89.3% zero-shot success rate. In this setting, zero-shot refers to deploying the learned policy without additional task-specific training on the real objects, suggesting that corrected demonstrations can serve as an effective bridge from human data to robot control.

Why it matters

The contribution is broader than a more accurate pose mapping. Contact preservation addresses whether the robot is interacting with the object in the intended way, while residual reinforcement learning addresses whether the resulting motion can actually be executed. Combining the two makes the demonstration-generation process better aligned with the requirements of sim-to-real learning.

The available material does not establish how the approach scales to substantially more hand designs, severe visual occlusion, or longer and more complex manipulation sequences. Its sensitivity to perception errors and contact-estimation quality also remains an important question for future evaluation.

Source: Hugging Face Daily Papers