Self Gradient Forcing Targets the Memory Gap in Long Video Generation

Lead

Long video generation is not simply about predicting the next frame. A model must preserve subject identity, background layout, scene structure and temporal rhythm over many generated steps. Recent autoregressive video diffusion systems often build on Self Forcing, where the student model trains on histories produced by its own rollout instead of always relying on ground-truth video context. This makes training closer to inference and helps reduce exposure bias.

The paper argues, however, that Self Forcing leaves an important weakness unresolved. Earlier generated latents are written into a historical key-value cache and later frames attend to that cache. Yet future-frame losses do not sufficiently supervise how those earlier latents should be encoded into useful keys and values. The authors call this the historical context-gradient gap.

Core ideas

• The cache is used, but not fully taught. In autoregressive video diffusion, the KV cache acts as causal memory. If it is treated only as frozen rollout state, future errors cannot effectively shape the way earlier context is written into memory.
• Self Gradient Forcing introduces a two-pass training process. In the first pass, the model performs a no-gradient autoregressive rollout that matches inference. At a sampled denoising exit step, it records both the self-generated context and the noisy latents provided to the model.
• The second pass reconstructs context gradients. The generated context is used as a stop-gradient clean-latent input, while the model recomputes the context KV representations and future-to-context causal attention. Losses on future video latents can then train the model to encode context into more effective causal memory.
• It avoids full serial backpropagation. SGF is not presented as an expensive end-to-end backward pass through the entire rollout. Instead, it restores the missing memory-writing supervision at a recorded denoising step, making the idea more practical for long-horizon training.

Why it matters

According to the paper summary, SGF outperforms Self Forcing in extensive long-horizon frame-wise and chunk-wise experiments under different initializations. The gains are especially visible in subject identity, background and layout consistency, and temporal stability. The authors also report that, using only a 5-second training window, SGF can extrapolate to videos lasting several minutes.

The broader contribution is conceptual as much as technical. It reframes long video generation as a problem of learning how to write durable causal memory, not just how to denoise or extend frames. If autoregressive video models are to scale to longer outputs, the quality of their historical memory representation will be central. Self Gradient Forcing offers one route to train that memory without requiring full backpropagation through very long generated sequences.

Source: Hugging Face Daily Papers