HandsOn Tests How Well Dexterous Robots Generalize

Introduction

A dexterous robot is not truly general-purpose if it can perform a skill only with one arm, one hand, or one familiar object. The harder question is whether the same behavior survives a change in embodiment, object identity, spatial relation, or task composition. HandsOn, a new simulation platform and benchmark, is designed to make that question measurable.

A benchmark built for cross-embodiment manipulation

HandsOn supports five robotic arms, six dexterous hands, and approximately 200 rigid, articulated, and deformable assets. It also includes about 18,000 curated human demonstrations in LeRobot v2.1 format, with synchronized front-view and wrist-camera observations. This combination gives researchers a common environment for comparing policies, action representations, and visual inputs across more than one robot configuration.

The benchmark contains 66 human-verified tasks divided into two tracks:

• Core-50 evaluates demonstration-based learning and task execution on a core collection of manipulation problems.
• Transfer-16 focuses on asset transfer, relation transfer, skill composition, and deformable manipulation, with no training demonstrations for the target tasks.

The split is important because success on familiar tasks can hide weak generalization. Transfer-16 asks whether a policy can reuse an acquired skill when the object, relationship, or task structure changes. That setting is closer to deployment, where a robot is unlikely to encounter exactly the same combination as in its training set.

What the results reveal

The authors evaluate diffusion policies, vision-language-action systems, world-action models, and prompted vision-language-model-based approaches. The best trained multi-task policy achieves 29.1% success on Core-50. Although this is a simulation result rather than a direct measure of real-world reliability, it indicates how challenging the benchmark is: the problem is not simply recognizing an object or producing a plausible trajectory. The policy must establish contact, coordinate several degrees of freedom, and preserve the achieved state over time.

Stage-level diagnostics make these failures more concrete. Policies struggle to establish reliable contact, coordinate palm and finger actions, and maintain goals after making partial progress. Composed tasks introduce another source of failure: the policy may execute individually reasonable skills in the wrong sequence. This suggests that dexterous manipulation needs more than a collection of local action patterns. It needs a control system that can connect perception, contact dynamics, state tracking, and sequencing.

The paper also reports a test with GPT-6 using move_eef for palm-and-finger control, without benchmark-specific fine-tuning. Across 660 trials, this setup reaches 5.2% on Core-50 and 1.9% on Transfer-16. These numbers should not be generalized to every vision-language model, but they do highlight the gap between high-level reasoning and dependable physical interaction. The project page additionally presents qualitative real-robot rollouts, offering evidence for sim-to-real exploration without implying broad real-world validation.

Why it matters

HandsOn provides a more informative comparison point than a single fixed-task success score. Researchers can inspect performance across embodiments, assets, relations, deformable objects, and composed skills. Its diagnostic breakdown is equally useful because it turns a failed trial into a more actionable question: was contact missed, were the fingers poorly coordinated, was the goal state lost, or was the sequence incorrect?

Future progress may require better contact modeling, hierarchical control for skill composition, cross-embodiment action representations, and closed-loop policies that handle deformation and uncertainty. With its code and PhysX demonstration dataset publicly available, HandsOn can serve as a shared testbed for studying where dexterous manipulation generalization still breaks down.

Source: Hugging Face Daily Papers