Spatial-Interactor Teaches VLMs Spatial Reasoning Through Interaction

Introduction

For a vision-language model, recognizing the objects in a scene is only the beginning of spatial understanding. An embodied system must also determine what changes after an object is moved, how a new viewpoint alters visible relations, and what the scene should look like after several actions. Spatial-Interactor targets this dynamic form of spatial reasoning.

From static relations to state transitions

Much existing spatial training is built around static images. Models may be asked to identify an object’s attributes or describe whether one object is to the left, right, in front of, or behind another. Such tasks establish useful spatial concepts, but they provide limited direct supervision for the changes produced by interaction.

Spatial-Interactor instead treats an interaction trajectory as a structured learning signal. A preceding observation, an action, and the subsequent observation form a local state-transition example. When multiple examples are connected, the model can learn dependencies across a trajectory rather than analyzing every frame independently.

A three-level curriculum

The framework divides learning into three stages:

• L1: Passive world-state transitions. The model learns how object motion, scene changes, and viewpoint shifts update the external world state.
• L2: Active self-state transitions. It models how its own actions or viewpoint changes alter the observations available to it.
• L3: Long-horizon interaction trajectories. It integrates consecutive local transitions while maintaining a coherent spatial state over an extended sequence.

To support these objectives, the authors introduce LSI-108K, a dataset assembled from simulated and real interaction trajectories. Its tasks are aligned with the different levels of the curriculum, covering both local transition understanding and long-range trajectory integration.

Two-stage training

For L1 and L2, the method applies supervised fine-tuning to establish reliable local transition modeling. The long-horizon stage then uses on-policy distillation. A privileged teacher branch receives segment-level transition descriptions and supervises the student’s on-policy chain-of-thought. The goal is to help the student connect successive changes into an ongoing spatial update process, rather than merely produce isolated explanations for individual observations.

Why it matters

The main contribution is a reframing of embodied spatial reasoning as state maintenance through interaction. By placing the observation-action-observation loop at the center of training, the framework offers a clearer bridge between visual understanding and physical-world behavior. According to the paper, experiments across multiple VLMs and spatial benchmarks show consistent gains in local transition modeling and long-horizon integration.

The available material does not include full benchmark scores, model-size comparisons, or results from deployment on physical robots. Spatial-Interactor should therefore be viewed as a training framework for dynamic spatial reasoning, not as a complete solution to general physical-world modeling. Its broader impact will depend on how well the approach transfers to less controlled and more open-ended environments.

Source: Hugging Face Daily Papers