Robot-Centric Pointmaps Help VLA Models See From the Robot’s Frame

Introduction

Vision-language-action models are becoming a central approach for building general robot policies. They take visual observations and language instructions as input, then predict the next robot action. Yet beneath this pipeline lies a subtle but important mismatch: the model usually sees the world through a camera, while the action it must output is defined in the robot’s own 3D coordinate frame.

The KAIST AI paper See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models addresses this mismatch directly. Instead of replacing existing VLAs with heavy 3D architectures, the authors propose a representation that injects robot-frame geometry into a familiar image-like format.

Key ideas

• The problem is a frame mismatch
  With a fixed camera, a policy can often learn a stable mapping from pixels to actions. But large robot datasets increasingly combine demonstrations captured with different camera positions and viewpoints. Under those conditions, the same robot-relevant spatial relation may appear very differently in the image, making generalization harder.

• Pointmaps store 3D coordinates in the robot frame
  A regular RGB image records color at each pixel. A robot-centric pointmap instead stores the 3D coordinates of the corresponding scene point in the robot’s coordinate frame. This gives the model information about where objects are relative to the robot, not merely where they appear in the camera view.

• The representation remains compatible with 2D VLAs
  A key design choice is to preserve the dense H×W grid expected by pretrained 2D VLA models. This makes the method easier to integrate than a fully new 3D pipeline. According to the paper summary, it can be added with one extra encoder and an element-wise addition step.

• The gains are strongest under viewpoint change
  On RoboCasa, robot-centric pointmaps improve both pi0.5 and SmolVLA and outperform representative camera-viewpoint and 3D-aware baselines. In real-robot tests, the advantage over an RGB-only policy becomes larger when the camera is moved to a placement not seen during training.

Why it matters

The contribution is not simply another larger robot model. It highlights that part of the VLA generalization problem may come from the input representation itself. A robot policy should care about where an object is relative to the gripper, base, or workspace; a raw camera view forces the model to infer that relationship indirectly.

Robot-centric pointmaps move some of this geometric alignment into the input, giving the policy a spatial description that is closer to the action space. This could be especially useful as the field moves toward larger and more heterogeneous robot datasets, where demonstrations are collected across labs, camera rigs, and deployment environments.

There are still practical assumptions. The system needs access to 3D geometry and a calibrated relationship between the camera and the robot frame. It is therefore best viewed as a pragmatic enhancement to current VLAs rather than a replacement for multimodal policy learning. Even so, the principle is compelling: if robot actions live in the robot’s coordinates, the model should be allowed to perceive the world in those coordinates too.

Source: Hugging Face Daily Papers