SANA-Video 2.0: Hybrid Attention for Faster High-Resolution Video Generation

Introduction

Video diffusion Transformers face a familiar scaling problem: as resolution, frame count, and duration grow, full softmax attention becomes increasingly expensive. SANA-Video 2.0 addresses this bottleneck by redesigning the attention stack rather than simply relying on larger hardware or heavier models.

The work introduces a unified architecture instantiated at 5B and 14B parameters. Its stated goal is to produce high-quality video up to 720p on a single GPU while staying close to the quality of full-softmax video DiTs. The central idea is to combine the long-sequence efficiency of linear attention with carefully placed softmax attention blocks that recover expressive token interactions.

Key points

• Hybrid Linear-Softmax Attention: Instead of applying quadratic softmax attention throughout the network, SANA-Video 2.0 uses gated linear attention for most token mixing. Periodic gated-softmax anchors are inserted at a 3:1 ratio, meaning softmax is used for roughly one quarter of the attention blocks. These anchors are designed to restore the full-rank token interactions that pure linear attention tends to miss.
• Attention residuals across depth: The paper introduces Block Attention Residuals, or AttnRes. Completed block summaries are routed into later linear layers, allowing the model to reuse the refreshed representations produced by the softmax anchors. According to the authors, this improves the effective rank of deeper layers by about 12%.
• Training from scratch: SANA-Video 2.0 is not presented as a converted or linearized version of a pretrained softmax model. It is trained from scratch with the hybrid structure in place, so the network learns the mixed attention pattern directly.
• Quality-efficiency trade-off: Reduced-resolution proxy studies identify 25% softmax attention as the preferred balance between quality and efficiency. This is the mechanism behind the 3:1 linear-to-softmax design.
• Reported performance: With 40-step sampling, the model reaches a VBench score of 84.30 in 13.2 seconds at 480p on a single H100. The compiled DiT forward pass is reported to be 3.2 times faster than a matched full-softmax baseline at 720p/60s, with the advantage increasing as video duration grows.
• System optimization: The authors also describe Sol-Engine, a full-stack optimization setup using kernel fusion, caching, and sparse attention. On top of the hardware-friendly backbone, it provides a further 3.58 times speedup. The 5B pipeline is reported at 13.06 seconds for 720p/5s and 120 times faster than Wan 2.2-A14B on one H100.

Why it matters

SANA-Video 2.0 is important because it offers a practical middle path between two extremes. Full softmax attention is expressive but costly; pure linear attention scales better but may lose important global interactions. By using softmax as periodic anchors and propagating their features through residual routes, the model aims to keep much of the quality while reducing compute.

If the reported results generalize, this design could influence future long-form and high-resolution video generation systems. It also reinforces a broader lesson in generative AI: progress is no longer only about parameter count. Architecture, training strategy, compiler-level execution, and inference-system engineering all shape what is feasible on real hardware.

Source: Hugging Face Daily Papers