Grounded Action Models Put 3D Object Grounding at the Core of Robot Control

Introduction

A robot cannot reliably perform an instruction such as “pick up the red cup” by understanding language alone. It must identify the correct object, locate it in 3D space, and determine how the end effector should approach it. Many current robot foundation models inherit their core representations from vision-language or video-generation systems. These backbones are powerful, but metric spatial grounding is not necessarily part of their pretraining objective. As a result, the ability to connect words with precise object locations is often learned only implicitly from robot demonstrations.

A paper introducing Grounded Action Models (GAMs) proposes a different design: make 3D grounding an explicit foundation for action prediction.

A shared representation for the selected object

GAM can be conditioned by natural-language instructions, point prompts, or box prompts. The input modality is converted into a common object-centric representation that identifies the selected target. This representation combines target-focused visual features with metric geometric information, giving the policy both an appearance-based description and a spatial account of where the object is located.

The object representation is then fused with the robot’s state history in a multi-stream Transformer. Rather than predicting only an isolated control command, GAM predicts action chunks, making the output better suited to continuous manipulation such as reaching, grasping, moving, and placing. The model can run autonomously, but it can also act as a low-level controller directed by a high-level planner. Because the planner can communicate through language, points, or boxes, the architecture supports a flexible division of labor: the planner decides what should happen, while GAM handles target-focused execution.

Reported results across simulated and real settings

On RoboTwin 2.0, GAM reaches a 55.3% average success rate across 50 tasks, compared with 52.0% for Spatial Forcing. Under scene randomization, it records 47.6%, versus 30.4% for Abot-M0. The result is notable because the action policy was trained only on clean-scene demonstrations, suggesting that explicit object geometry can help transfer behavior to altered visual conditions.

On LIBERO-PRO, GAM reports a 61% average success rate across 16 perturbation settings, compared with 53% for π₀.₅. The largest gains appear when the target is relocated or a new target is designated, precisely the situations in which a policy must identify the current object rather than rely on a fixed scene layout.

The real-robot tests follow the same pattern. On a bimanual YAM under visual shift, GAM retains 17 successes out of 20, while π₀.₅ achieves 4 out of 20. When combined with a Molmo2 planner on a Franka robot, GAM reaches 64.7% in-distribution and 49.8% out-of-distribution step completion on long-horizon, memory-dependent tasks.

Why the approach matters

GAM reframes the relationship between planning and control. A high-level system can specify the object or goal, while a grounded action model translates that specification into spatially appropriate movements. This is potentially more robust than asking a general-purpose policy to infer object identity, location, and control geometry entirely from demonstrations.

The reported evidence is still limited to the paper’s benchmark tasks and a small set of real-robot evaluations. It does not establish that the approach solves difficult open-world manipulation, including severe occlusion, novel object categories, delicate contact, or long sequences with frequent failures. Even so, GAM offers a clear design lesson for embodied AI: explicit, measurable spatial grounding may deserve to be treated as a first-class component of robot foundation models, alongside language and visual representation.

Source: Hugging Face Daily Papers