SLA Brings 3D Spatial Learning into Continual Robot Pre-training

Introduction

A robot that is asked to pick up a cup must do more than recognize the object and understand the instruction. It must estimate where the cup is, how far away it sits, and how the arm should approach it. Yet many vision-language-action (VLA) models inherit their visual representations from two-dimensional vision-language pre-training. Spatial supervision is often added only when adapting a model to a particular downstream task, making 3D understanding more like a local skill than a transferable foundation capability.

SLA, short for Spatial-Language-Action Model, proposes to change that training recipe. The method brings spatial alignment into continual pre-training on robot experience and aims to preserve it during downstream post-training.

Key ideas

• Joint action and spatial objectives. SLA starts from a pretrained pi05 policy. It continues to predict robot actions while aligning intermediate visual embeddings with spatial representations produced by a frozen 3D perception model. The goal is to make spatial structure part of the model representation rather than a task-specific add-on.
• A unified view of heterogeneous data. The authors curate robot datasets covering different embodiments, tasks, and scenes. Because geometric annotations are uneven across datasets, the teacher can use RGB observations together with partially available depth and camera calibration.
• Adaptive multi-layer fusion. Spatial cues may be distributed across different VLA layers. Feature-dependent gates adaptively aggregate visual embeddings from multiple layers before spatial supervision is applied, instead of selecting one fixed layer for every example.
• Less reliance on demonstrations. Probabilistic state dropout reduces the model’s dependence on state inputs and demonstration trajectories during training. This encourages greater use of visual evidence and is intended to mitigate trajectory overfitting.
• Training-time assistance only. The 3D teacher and other auxiliary components are not required at inference time. Once training is complete, the resulting model can reuse the pretrained VLA deployment pathway without adding a separate spatial teacher.

Why it matters

The central contribution of SLA is a change in where spatial competence is learned. Rather than treating geometry as a narrow adaptation target, it attempts to make spatial understanding a transferable representation across tasks and robot platforms. For manipulation involving precise grasping, placement, or contact, this could help connect scene interpretation with action selection in 3D space.

The work also highlights a practical challenge in scaling embodied models: robot datasets rarely share the same sensors, camera parameters, or depth availability. A model therefore has to learn a consistent geometric representation from incomplete and uneven supervision. SLA addresses this through gated feature fusion and flexible use of geometric inputs, but the extent of its generalization should be assessed through the paper’s experiments and broader real-robot testing. Overall, it presents continual pre-training as a route from 2D semantic competence toward more spatially grounded action models.

Source: Hugging Face Daily Papers