GAE Builds a Geometry-Native Latent Space for 3D-Consistent World Generation

Introduction

Video generators have become highly effective at producing realistic frames, but realism at the frame level does not guarantee a stable 3D scene. As the viewpoint changes, object shapes, spatial relationships, or camera motion may drift. GAE, short for Geometry-Native Autoencoder, approaches this problem from a different angle: the weakness may lie not only in the generator, but also in the latent space where generation takes place.

Key ideas

• Geometry is embedded in the latent itself. Many visual generation systems evolve latents that primarily organize color, texture, and local appearance. GAE reparameterizes features from a geometry foundation model into a compact representation that preserves semantic and cross-view structure.
• One latent state supports multiple observations. The GAE latent can be jointly decoded into RGB appearance, depth, camera information, and 3D point maps. The model therefore represents a unified scene state rather than a collection of unrelated frames.
• It works with a conventional conditional generator. On top of this geometry-native representation, the authors use a conditional flow model. Camera trajectories provide a control signal, linking the generated video to an intended pattern of viewpoint motion.
• The evaluation measures more than image quality. With the generator and training protocol held fixed, replacing the baseline latent with GAE lowered FVD by 12.7% on RealEstate10K and 23.1% on DL3DV. Camera-trajectory error on RealEstate10K was reduced by half.

Why it matters

The central contribution is not an additional depth prediction head attached to a video model. It is a change in the model’s internal interface. When the latent already encodes geometric relationships across views, the generator has a better chance of preserving those relationships while synthesizing photorealistic appearance. GAE consequently functions as a bridge between perception and generation: geometry-aware perception supplies the structured representation, while a generative model produces diverse outcomes within it.

This design is relevant to world models, controllable video synthesis, and novel-view generation. Several questions remain open. Can the latent scale to longer videos and more complex scenes? Does joint decoding impose a trade-off with texture fidelity or sample diversity? And how well do current geometry metrics reflect what users perceive as temporal and spatial consistency? The reported results nevertheless support a broader lesson: the latent space is not merely an implementation detail. It can be a primary determinant of whether a generator produces a sequence of attractive images or a coherent three-dimensional world.

Source: Hugging Face Daily Papers