Back to articles

ZeroVLN Navigates Without Navigation-Specific SFT of Its VLM Brain

3 min read

Vision-language navigation (VLN) requires more than understanding an instruction. An agent must repeatedly interpret its surroundings, choose where to go, control movement, and adjust after receiving feedback. Many approaches pair a vision-language model (VLM) with a learned action module and use navigation data to fine-tune the VLM to predict geometric goals. ZeroVLN proposes a different division of labor: keep the model’s “brain” from having to relearn navigation, and let it express movement decisions through natural-language tool calls that a specialist action model translates into continuous motion.

Key points

  • Separate semantic decisions from motor execution. The VLM proposes a movement request based on the task and visual scene. An action expert turns that request into continuous control. Rather than requiring the VLM to directly predict geometric goals, the design gives it a semantic interface to the executor.
  • Train execution first, then optimize in a loop. The researchers build reference trajectories from movement requests generated by the base VLM. They supervise only the action prefix that is carried out before the next replan. ZeroVLN then applies closed-loop agentic reinforcement learning to improve both the VLM’s tool calls and the expert’s continuous actions.
  • Use feedback across the trajectory. The method combines process and outcome rewards and uses a shared trajectory-level advantage to update both policies, while keeping separate gradient paths for the brain and the action expert. “Zero navigation-domain SFT” does not mean the VLM is never trained: the paper avoids that form of supervised adaptation, but later jointly optimizes the policies through closed-loop RL.
  • Report navigation gains with limited weight change. Using the same backbone and training episodes, the paper says ZeroVLN achieves higher success rates than NaVILA and DualVLN on all three standard VLN-CE benchmarks. It reports 0.01497% relative displacement of the VLM’s weights and says the base model’s general visual capabilities are largely preserved.

Why it matters—and what remains unclear

The contribution is not simply another action module. It is a change in how a multimodal model and a controller communicate: the VLM conveys what movement is needed in readable semantic terms, while the expert handles continuous execution. This arrangement may reduce the need to alter a general-purpose model for a specific navigation task, while allowing decisions and control to improve through interaction. But the supplied material does not include benchmark scores, detailed evaluation settings, or results across a broader range of environments. It therefore does not establish the size of the reported advantage or its limits. The cautious takeaway is that ZeroVLN demonstrates a way to train hierarchical navigation while retaining most of the VLM’s original weights; the broader practical value needs confirmation from the full paper and further evaluations.

Source: Hugging Face Daily Papers

Comments

Checking sign-in status...

Loading comments...

Related articles