Qwen-Drive-1.0: Extending Vision-Language Models to Autonomous Driving

Autonomous-driving systems have traditionally separated perception, scene understanding, and planning into specialized components. Qwen-Drive-1.0 explores a different direction: keeping the architecture of a pretrained vision-language model while connecting its shared representations to 3D driving perception and trajectory generation.

What the system adds

The paper’s contribution is a unified task setup rather than a single isolated prediction head:

• An explicit 3D interface. An external bird’s-eye-view, or BEV, perception head jointly performs 3D object detection, semantic occupancy prediction, and BEV map segmentation. Besides serving practical perception tasks, the head acts as a probe into the 3D information contained in the shared representations.
• A dedicated planning expert. The Planning Expert conditions on the shared VLM features and generates future ego trajectories. This connects visual-language understanding with a continuous output that is directly relevant to vehicle motion.
• A staged training recipe. Driving supervision is combined with general-purpose vision-language data. The stated goal is to acquire driving-specific competence without discarding broad visual understanding and instruction-following abilities learned during pretraining.

Why this matters

Driving requires more than recognizing objects in an image. A useful model must reason about spatial layout, road structure, occupancy, and how the current scene constrains future movement. By exposing BEV predictions for detection, occupancy, and mapping, Qwen-Drive-1.0 provides a relatively inspectable route for testing whether a shared VLM representation contains useful 3D information.

The approach also addresses a common tension in domain adaptation. A model tuned too narrowly for driving may lose its ability to answer questions or interpret general visual content, while a general-purpose VLM may lack the spatial precision needed for planning. The staged recipe attempts to preserve both sides rather than treating driving as a complete replacement for general vision-language learning.

According to the supplied material, the model shows strong 3D perception and driving-scene understanding while largely retaining general vision-language capability. Its planning performance is evaluated in open-loop, pseudo-closed-loop, and closed-loop settings, where it is reported to be highly competitive. These results are important less as a final verdict than as evidence that a general VLM can serve as a foundation for multiple driving functions.

What remains open

The paper describes Qwen-Drive-1.0 as an initial step, so the approach should not be read as a complete solution to safety or deployment. Questions remain about robustness in long-tail situations, the stability of shared representations under complex traffic interactions, and how well closed-loop behavior transfers across environments. Still, the work outlines a promising recipe: retain a general VLM, expose its spatial knowledge through an inspectable BEV interface, and attach a specialized planner for embodied action.

Source: Hugging Face Daily Papers