Long-context inference is expensive not only because it processes many tokens, but also because attention may spend compute on irrelevant content. Mixture of Block Attention (MoBA) reduces that work by selecting KV blocks for each query, and FlashMoBA improves its GPU execution. Yet a fixed Top-k budget gives every query the same retention limit, even when score distributions differ. Query-range-based task ownership can also make separate tasks load the same KV block more than once.
LightningMoBA addresses these issues by designing routing and GPU execution together, rather than treating sparse selection as an isolated change.
Three design choices
- Score-threshold routing: Instead of requiring every query to select a fixed number of blocks, LightningMoBA uses a threshold on block scores. Queries can therefore retain different numbers of blocks. Sparse metadata is emitted during scoring, without maintaining an ordered Top-k candidate set or training an extra indexer.
- KV-centric task ownership: Within an attention head, query rows that access the same KV block are grouped together. A task can reuse the K/V tile it loads. When a block attracts many queries, its work is split into bounded tasks to limit the workload assigned to each one.
- Pipelined execution: On Hopper GPUs, the implementation overlaps regular K/V transfers and irregular query gathers with Tensor Core computation, aiming to reduce stalls.
The separation is important: routing determines how much each query retains, while the execution backend organizes that variable work around data reuse and bounded task sizes.
What the experiments show
Across five checkpoints and context lengths from 4K to 128K on the full RULER suite, the researchers matched average retention to a Top-16 budget. Compared with FlashMoBA, LightningMoBA delivered 1.20–1.60× end-to-end generation speedups, while the mean RULER score rose from 74.42 to 75.28. To isolate routing from backend changes, the paper also compared both routing strategies on the same optimized backend. Across six configurations on two models, dynamic routing at a Top-64 budget achieved 1.16–1.56× end-to-end speedups and RULER gains of 0.85–5.95 percentage points.
The training test used Qwen3-4B with a 128K context for 100 steps. Both routing approaches shared the optimized forward implementation and an adapted backward implementation. Dynamic routing reported end-to-end training speedups of 1.23× and 1.60× at Top-32 and Top-64, respectively, while cross-entropy remained close between the strategies.
The broader lesson is that sparse attention performance depends on more than skipping arithmetic: metadata generation, memory reuse, and GPU scheduling also matter. The results support efficiency and quality gains in the configurations evaluated, but do not establish performance for every serving workload. Production systems would still need to assess factors such as batching behavior and tail latency. LightningMoBA is therefore best read both as a specific MoBA optimization and as a design example for implementing dynamic sparse attention on GPUs.
Source: Hugging Face Daily Papers
Comments
Checking sign-in status...
Loading comments...