DriveZero: Teaching Autonomous Driving Beyond Human Demonstrations

Introduction

Many end-to-end autonomous-driving systems are trained by imitating human driving logs. This provides a practical source of supervision, but it also imposes a clear ceiling: the model can only reproduce behaviors, routes, and intentions represented in the recordings. Rare situations and reasonable alternatives that human drivers never demonstrated are difficult to learn through imitation alone. DriveZero, introduced by Xiaomi’s autonomous-driving team, addresses this limitation by generating supervision through interaction rather than relying exclusively on recorded actions.

A division between seeing and acting

DriveZero separates driving into a perception model and an action model. The authors argue that the two components need different training regimes. Perception benefits from large and diverse visual corpora, while action learning requires closed-loop feedback, where the consequences of a decision affect what happens next.

• DriveRL for action learning. DriveRL converts real nuPlan logs into interactive mixed-agent worlds. Log replay, rule-based controllers, and learned policies can coexist as behavior providers for different background actors. This creates a more flexible training environment than simply replaying a fixed scene. A privileged teacher policy with 5.7 million parameters is trained with PPO through closed-loop rollouts, allowing its decisions to respond to the evolving environment.
• DriveVFM for visual understanding. DriveVFM combines frozen capabilities from DINOv3, SigLIP2, SAM, and Depth Anything V2 into a single driving backbone. It learns from raw images without task-specific annotations, aiming to provide broad representations of objects, semantics, geometry, and depth.
• DriveZero through distillation. The final camera-only planner encodes multi-view images with DriveVFM and decodes trajectory proposals. Instead of copying human actions directly, it learns from trajectories rolled out by the DriveRL teacher. Because the teacher is goal-conditioned, it can also be queried with augmented driving intents, producing goal-consistent supervision unavailable in the original logs.

Results and caveats

On nuPlan, DriveRL with value-guided test-time action search reports a mean score of 93.57 across the Val14, Test14-hard, and Test14-random community splits in both non-reactive and reactive settings, exceeding the log-replay expert. DriveZero, trained only on DriveRL rollouts, is reported to achieve state-of-the-art results on NAVSIMv1, NAVSIMv2, and the closed-loop HUGSIM benchmark.

The results suggest that closed-loop reinforcement learning can complement human data by generating behaviors that logs do not contain. Still, the privileged teacher has access to information unavailable to a camera-only vehicle, and the fidelity of mixed-agent simulators remains important. Benchmark gains should therefore not be interpreted as proof that long-tail safety has been solved in real traffic.

Why it matters

DriveZero presents a training blueprint rather than just another planner: use large-scale visual pretraining to improve world understanding, use closed-loop reinforcement learning to improve interaction, and use distillation to combine both abilities in a deployable end-to-end system. If the approach transfers reliably to broader real-world conditions, it could reduce dependence on costly human demonstrations and make it easier to generate data for new driving goals and rare behaviors.

Source: Hugging Face Daily Papers