VideoRAE Recasts Video Foundation Models as Generative Latent Spaces

Lead

Modern video generators depend heavily on the latent space in which they operate. Many systems compress video with 3D variational autoencoders before training diffusion or autoregressive generators on the resulting representations. But conventional 3D-VAEs are usually trained to reconstruct pixels, which does not necessarily mean their latents preserve high-level semantics, motion patterns, or spatio-temporal structure.

The arXiv paper VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders addresses this latent-space problem directly. Instead of asking a VAE to learn video structure from scratch, it asks whether frozen video foundation models can provide a better basis for generative modeling.

Key points

• From pixel latents to representation latents: VideoRAE leverages frozen video foundation encoders such as V-JEPA 2 and VideoMAEv2, aiming to reuse their video understanding capabilities for generation.
• Multi-scale hierarchical features: The method extracts hierarchical features from the frozen encoder, preserving richer semantic and temporal information than a purely pixel-oriented compression objective may capture.
• Lightweight 1D self-attention projector: These features are compressed through a lightweight 1D self-attention module, producing compact latents suitable for downstream generators.
• Two generative regimes: VideoRAE supports continuous latents for Diffusion Transformers and discrete tokens for autoregressive models through multi-codebook high-dimensional quantization.
• Representation alignment during decoding: The decoder is trained with local and global alignment against the frozen VFM teacher, improving semantic preservation and avoiding the need for KL regularization.

Why it matters

According to the abstract, VideoRAE performs strongly in both continuous and discrete reconstruction settings. On UCF-101 class-to-video generation, it reports gFVD scores of 40 with autoregressive generators and 93 with DiT generators, described as state of the art in the paper. It also converges about five times faster than competing autoencoder baselines. In a controlled 2B-scale text-to-video study, replacing LTX-VAE with VideoRAE led to faster convergence under comparable conditions.

The broader implication is that video foundation models may serve not only as perception or understanding backbones, but also as latent-space providers for generative systems. If their frozen representations can be made compact and reconstructable, future video generators may benefit from semantic structure already learned by large-scale video models.

The evidence is still based on the paper’s reported results, and the model and code are described as forthcoming. The next questions are whether VideoRAE scales to broader open-domain video, longer sequences, and diverse text-to-video workloads beyond the controlled setting reported here.

Source: arXiv