M⁴World: A Controllable Multimodal World Model for Driving Simulation

Lead

Autonomous-driving simulation is moving beyond replaying logged data toward generating interactive, editable worlds. M⁴World, a new paper on arXiv, presents a multi-view and multimodal generative driving world model that aims to produce future surround-view video streams and synchronized LiDAR scans. Its central promise is not only realism, but controllability: users should be able to decide where individual objects appear, how they look, and whether they remain consistent over extended rollouts.

Key points

• Multi-view and multimodal generation: M⁴World targets the sensor setup commonly used in autonomous driving. It synthesizes surround-view video and aligned LiDAR scans, making it more relevant to perception pipelines than a video-only generator.
• Object-level manipulation: The model introduces a flexible conditioning interface for explicit control over both spatial layout and visual appearance of individual objects. This is important for testing targeted scenarios rather than relying on broad prompts.
• Minute-long streaming: Long video generation often suffers from drift, identity switches, or gradual scene collapse. The paper describes a multi-stage training framework that supports online causal generation in four denoising steps while maintaining coherent dynamics during extended rollouts.
• Customization for rare cases: M⁴World adds efficient few-clip post-training and visual-reference-conditioned generation models. The goal is to preserve general generation ability while enabling customization for long-tail objects and unusual driving cases.
• Evaluation beyond realism: The authors introduce an automated VLM-based judging pipeline to assess scene-level condition following, view-wise object controllability, and cross-view object consistency.

Why it matters

For autonomous driving, the most valuable simulated data is often not ordinary traffic, but rare, risky, and hard-to-collect edge cases. A world model that can generate stable, editable, multi-sensor driving sequences could become useful for simulation testing, long-tail data augmentation, and controlled scene editing. M⁴World is notable because it frames generation quality and operational controllability as equally important goals.

That said, the available material is an arXiv abstract and paper page, so the full technical claims still depend on the complete experimental details: dataset coverage, evaluation design, comparisons with prior systems, and whether generated LiDAR provides measurable downstream benefits. Even with that caveat, M⁴World points to a clear direction for driving world models: from short, visually plausible clips toward long-horizon, controllable, multimodal simulated worlds.

Source: arXiv