SparsePR Speeds Up Video Attention with Partitioning and Residual Reconstruction

Long spatiotemporal sequences make attention one of the major inference costs in video generators and world models. Block-sparse attention can reduce the number of query-key interactions, but selecting blocks is harder than simply keeping those with the highest apparent attention mass. Queries assigned to a common route may still have poorly overlapping supports, and retained mass alone does not predict the error introduced after softmax when other interactions are skipped.

SparsePR addresses this problem with two linked ideas: make the partition more representative of attention responses, then reconstruct the contribution that was omitted.

• Response-coupled partitioning: The method samples query rows and examines their responses to keys. It forms paired K/V groups and uses their response centroids to create query-response coordinates. Routing is therefore based on how queries respond to the key-value structure, rather than only on positions or independent row-wise scores.
• Probe-fitted residual reconstruction: A small set of query rows is evaluated with exact attention. The difference between the exact and sparse outputs becomes a probe residual. SparsePR fits a call-specific affine correction from the sparse output, restricting the correction to the output subspace observed in those residuals.
• Hardware-oriented sparsity: Instead of treating retained attention mass as the main objective, the method targets attention-reconstruction error and includes a block-sparse GPU implementation intended to turn algorithmic sparsity into practical speed.

The paper evaluates SparsePR on four heterogeneous systems: HunyuanVideo, Wan2.2, Cosmos2.5, and Cosmos3-Nano. At a realized executed-pair density of 22.0–26.0%, the method preserves generation quality and achieves reported end-to-end speedups of 1.48x–2.61x. Ablations indicate that probe fitting accounts for most of the reduction in reconstruction error. Response-coupled partitioning contributes by lowering hard-drop error and improving reconstruction when the number of exact probes is limited.

The broader contribution is a shift in how training-free sparse attention is framed. The goal is not only to identify important blocks, but also to estimate the output effect of the blocks that are deliberately skipped. SparsePR performs this calibration at inference time without changing model parameters, using a small amount of exact computation to infer the current attention call’s residual structure. Its practical benefit will still depend on probe budgets, partition quality, and GPU kernel efficiency, but the approach offers a concrete route for accelerating long-context video generation and world-model inference.

Source: Hugging Face Daily Papers