KVAE: A Tokenizer Family for Audio, Image and Video Generation

Introduction

Latent diffusion models depend heavily on the quality of their tokenizer. Before a model can learn to generate an image, a video or an audio clip, the raw signal must be mapped into a compact latent representation. That representation affects training efficiency, reconstruction fidelity and the eventual quality of generated samples. The KVAE technical report from Kandinsky Lab focuses on this often underappreciated layer of the generative stack.

Rather than presenting a single model, the report introduces a family of tokenizers built for text-conditioned generation across multiple modalities: audio, image and video. The goal is not only to compress data, but to produce latent spaces that are useful for downstream latent diffusion models.

Key points

• A multimodal tokenizer family: KVAE consists of KVAE-Audio, KVAE-3D and KVAE-2D, covering audio, video and image inputs respectively.
• Audio tokenizer: KVAE-Audio is a continuous full-band 48 kHz tokenizer with a 50 Hz latent rate and 64 latent channels, designed for subsequent audio generation.
• Video tokenizers: KVAE-3D includes two causal video tokenizers with 4x16x16 and 4x8x8 compression settings. The causal design is particularly relevant for temporal generation, where future frames should not leak into past representations.
• Image tokenizer: KVAE-2D compresses images by a factor of 8 and uses 32 latent channels, making it a candidate foundation component for image latent diffusion systems.
• Broad evaluation: The report covers reconstruction metrics such as PSNR, LPIPS and PESQ, as well as generation-oriented metrics including Frechet Distance, CLIP score and CLAP score. It also reports subjective side-by-side comparisons.

Why it matters

Tokenizers are increasingly becoming a competitive layer in multimodal AI. A strong diffusion backbone cannot fully compensate for a poor latent representation: excessive compression can remove crucial detail, while inefficient latents can make training slower and more expensive. KVAE addresses this layer directly and across modalities, which makes the work relevant beyond any single generation task.

According to the report, KVAE matches or surpasses leading open-source tokenizers associated with systems such as Wan-2.2, HunyuanVideo-1.5, FLUX.2, MovieGen, StableAudio and MMAudio on a mix of objective and subjective evaluations. Those claims should be read in the context of the paper’s experimental setup, but the release is notable because the authors also share inference code, model weights, training details, model selection methods and ablations on design choices.

For the open-source community, this makes KVAE useful as both a practical component and a reference point. Developers building text-to-audio, text-to-image or text-to-video systems can inspect how the tokenizers are configured and benchmark them against their own pipelines. More broadly, the release reinforces a trend: future multimodal generation may be shaped as much by representation design as by the diffusion model itself.

Source: Hugging Face Daily Papers