NEO-VLA Lets Robot Policies Act on Evolving Visual Representations

Introduction

Vision-language-action models are often built as a pipeline. A vision encoder first turns an image into a compact representation, a language model combines it with an instruction, and an action module then predicts how a robot should move. The separation is practical because it makes it possible to reuse established vision-language systems. Yet it may also remove the details that matter most for manipulation: object boundaries, local geometry, depth relationships, and small changes in state.

The paper From Pixels to Actions - Towards Native Vision-Language-Action Policies presents NEO-VLA, a framework that questions whether action policies should inherit only the final output of a visual encoder.

Key ideas

• No fixed visual endpoint. NEO-VLA uses lightweight patch-convolution layers to tokenize images, but does not send those tokens through a separate visual encoder whose final representation becomes the only input to control. Image and language tokens evolve together through the backbone.
• Access to representations at multiple depths. A flow-matching action expert can use instruction-conditioned visual features from different stages of the network. The policy therefore does not have to rely solely on a highly abstract final representation.
• Architecture matters beyond encoder removal. In controlled comparisons, the native design outperforms encoder-based alternatives. Full fine-tuning narrows the difference but does not eliminate it, while increasing model scale alone is not enough to recover the same advantage.
• Useful control signals may peak early. Layer-wise probing suggests that action-relevant information can be strongest before the final visual representation. For manipulation, deeper processing is not automatically better if it discards spatial or state-sensitive details.

Why it matters

NEO-VLA’s contribution is not simply the removal of a vision encoder. It proposes a different interface between perception and control. A representation optimized for recognizing objects or describing scenes may be good at answering “what is visible,” but less useful for deciding “where should the gripper move next.” Robot control needs semantic understanding as well as continuously changing spatial and state information. Giving the action policy access to the visual computation hierarchy may therefore be more valuable than only enlarging the perception module.

The framework shows stronger control across diverse manipulation settings and visual distribution shifts, indicating that native representations can improve robustness to changes in appearance. The study also reports clear limitations: shifts in spatial configurations and task goals remain challenging. Intermediate visual access is not a complete solution to generalization; spatial reasoning, goal interpretation, and flexible control still need to improve.

More broadly, the work shifts the design question from “how should vision be compressed?” to “which parts of visual computation should remain available to action?” For embodied intelligence, perception and control may be most effective when they cooperate across several layers rather than being separated by a single fixed feature interface.

Source: Hugging Face Daily Papers