HumanoidToolBench Tests Tool Use from Selection to Mobile Execution

Introduction

Tool use gives humanoid robots a way to overcome the physical limits of their bodies. A broom can extend their reach, while a gripper or other implement can change what the end effector is able to do. Yet tool use is not simply a matter of grasping an object. A robot must infer what the task requires, select an appropriate option from a set of candidates, manipulate it correctly, maintain a stable posture, and sometimes move through the environment before the task is complete.

Many existing robotics benchmarks isolate one of these abilities. They may measure grasping, object interaction, or a short-horizon manipulation skill, but not the complete sequence on a humanoid platform. HumanoidToolBench, introduced by the SNU PI team, is designed to address that evaluation gap.

Key points

• A benchmark for the full pipeline. The suite contains 18 tasks spanning three scenarios, three execution levels, and two tool-set modes. This structure makes it possible to distinguish selection from manipulation and from more demanding execution that includes locomotion.
• Simulation and hardware data. The accompanying ToolBook dataset contains roughly 3,100 demonstrations collected in simulation and on a real Unitree G1. It is intended to support both policy development and comparable evaluation.
• Correct selection does not guarantee success. The authors evaluate seven policies in simulation and three policies on the real robot. The results show a substantial gap between identifying a suitable tool and completing the task. A policy may make the right choice but fail during grasping, control, or movement.
• Generalization remains fragile. Additional probes with GR00T N1.7 report lower selection accuracy for unseen tools. The system may also continue executing an existing task when given unrelated instructions, pointing to limitations in instruction handling and behavioral control.

Why it matters

The main contribution is not merely a larger list of tasks. HumanoidToolBench evaluates “what to use” and “how to use it” within one chain of decisions. On a humanoid, tool use can involve visual understanding, semantic matching, arm coordination, whole-body control, and locomotion. Measuring only one stage can make a system appear more capable than it is in practice. Separating the stages, while still requiring end-to-end completion, offers a clearer way to identify whether failure originates in tool recognition, action planning, or physical execution.

The benchmark also highlights why familiar-object success rates are insufficient for general-purpose robot models. Unseen tools, changing tool sets, and unrelated instructions create uncertainty closer to real deployment. A robust system should connect selection and execution through feedback: if the grasp is unstable, the tool is ineffective, or the task state changes, the robot should revise its decision instead of blindly following the original plan. With public code and data, HumanoidToolBench provides a reproducible starting point for studying that capability.

Source: Hugging Face Daily Papers