L3G-World Uses Latent 3D Gaussian Memory for Real-Time Infinite Worlds

Introduction

Interactive world models must keep producing visual content as a user moves through a scene. Long-horizon exploration makes this difficult for two reasons. A model that relies mainly on temporal context can lose spatial consistency over time, while an explicit 3D memory may grow with every generated segment. Existing 3D-memory pipelines also commonly render the memory into RGB and encode that image again before conditioning a latent video generator. The repeated conversion adds a substantial real-time cost.

L3G-World, introduced by a KAIST team, addresses both issues with a Latent 3D Gaussian Memory, or L3G. The representation uses compact 3D Gaussians to retain scene structure, while also attaching a VAE latent feature to each Gaussian. As a result, the memory can be rendered directly into the latent space expected by a video diffusion transformer, rather than passing through an RGB reconstruction and a second VAE encoding step.

Key points

• Growth follows scene extent: A memory updater creates Gaussians in newly observed regions and refines existing ones. Memory therefore grows mainly with the area explored, rather than simply with the length of the generated stream.
• Latent-space rendering: Storing VAE features inside the Gaussians allows the memory to condition the DiT directly, removing a costly RGB-to-latent conversion stage.
• Alternating autoregressive pipeline: The updater and the video diffusion transformer take turns. The updater expands or improves the scene memory, while the DiT generates the next visual content conditioned on that memory and the user’s actions.
• Reported real-time performance: The paper reports generation at up to 9 FPS. Compared with the most efficient prior 3D-memory baseline for each metric, it reports up to 38.6 times fewer 3D primitives, 3.2 times faster memory updates, and 359 times faster memory rendering.

Why it matters

The central contribution is not simply a faster renderer. L3G-World connects persistent spatial memory and latent video generation in the same representation pipeline. That design could be useful for world models that need to revisit previously explored areas, maintain coherent geometry across long trajectories, and continue expanding into unseen regions without storing every generated frame as a separate memory entry.

The supplied material does not specify how performance changes across scene categories, hardware configurations, or more complex interaction patterns. The 9 FPS figure should therefore be read as a result under the paper’s experimental setup, not as a universal deployment guarantee. Even with that limitation, the approach presents a practical direction for interactive world models: compact the 3D memory, attach the representation needed by the generator, and avoid repeated image-space conversion.

Source: Hugging Face Daily Papers