Breaking Robot Vision Shortcuts with Latent Interface Training

Robot foundation models are increasingly expected to translate visual observations directly into actions. Yet strong performance on familiar data does not necessarily mean that a model has learned the right visual concepts. It may instead exploit correlations that happen to hold in the training distribution.

For example, a particular camera view, background, lighting pattern, or object appearance may repeatedly co-occur with a demonstrated action. If the model treats that cue as evidence for the action, it can appear competent during in-distribution evaluation while failing when the environment changes. These vision-action shortcuts are especially problematic for robots, because deployment often involves new viewpoints, illumination conditions, and distractors.

Researchers from the National University of Singapore’s MAGIC Lab and collaborators propose Latent Interface Training, or LIT, as a way to constrain this interaction. The framework is organized into two stages rather than asking the action model to learn visual grounding and control simultaneously from the beginning.

Stage one: establish an image-free action prior.

The action expert receives a language instruction, robot state, and the terminal SE(3) pose of the end effector for each demonstrated action chunk. It learns to generate a chunk that moves toward a specified spatial goal without access to images. This gives the expert a goal-directed action prior that is not based on background appearance or other visual coincidences.

Stage two: introduce a controlled latent interface.

Visual and semantic representations are then aggregated by a latent interface. This interface is the pretrained action expert’s only route for receiving visual conditioning. It is also trained to reconstruct the terminal pose used in stage one. The pose objective encourages the interface to preserve spatial information that matters for action generation, while making it harder for unrelated visual details to flow directly into the action module.

The distinction is important. LIT does not argue that robots should ignore vision. Visual input remains necessary for locating objects, estimating relationships, and responding to the environment. Instead, the method attempts to make visual conditioning more structured: information should influence action through a representation that retains task-relevant spatial goals.

The paper evaluates LIT with four vision-language-action or world-action architectures: Pi0.5, MolmoAct2, FAST-WAM, and ImageWAM. The reported results show gains of 3.87 to 10.70 percentage points in overall LIBERO-Plus success, while average LIBERO performance is preserved or improved. In real-robot evaluations covering three tasks, performance aggregated across unseen camera configurations, lighting variations, and distractors improves by 13.30 to 16.70 percentage points.

The broader implication is that robot generalization depends not only on the size of a visual encoder or the amount of data, but also on the interface between perception and control. Explicit spatial supervision offers a way to preserve useful visual information while reducing shortcut learning. At the same time, the available summary does not provide a complete breakdown by task, architecture, or perturbation type, so the method’s limits require examination of the full paper and further deployment studies.

Source: Hugging Face Daily Papers