Inference & Serving

DoSA reduces the cost of finding sparse-attention keys by ruling out candidates before their full indexer scores are computed. The method preserves exact Top-K selection rather than trading accuracy for speed.

Elastic Latent Recurrence, or ELR, is designed to let one language model trade inference compute for quality without sacrificing its one-loop baseline. The paper combines recurrent-state gating, mixed-depth training, and loop-aware conditioning to make extra passes useful and deployable.

vLLM has added bidirectional KV transfer to its Prefill-Decode serving design, allowing prefill workers to reuse context retained by decode workers on later turns. The approach reduced TTFT by up to 3.3x in the reported long-context benchmarks.

VLA acceleration cannot be judged by latency and average success alone: small action deviations may compound into failures that the original policy would have avoided. CARE selects the fastest candidate that satisfies a statistically controlled risk budget.

TokenRouter is a serving system for LLM routing at token granularity. It addresses execution desynchronization and batch admission delays with asynchronous model subservers and delayed batching.

SparseDecoding targets two practical gaps in LLM inference: pruning calibration often uses data unlike the model’s own generated tokens, while existing sparse kernels frequently emphasize SpMM rather than decoding-heavy SpMV. Its decoding-aware method and N:M sparse kernel deliver up to 1.48× end-to-end decoding speedup on A100 GPUs.

SparseEngine is a sparse-first inference engine designed to make heterogeneous KV-cache methods easier to deploy in long-context LLM agents. Its authors report substantial gains in throughput and decoding speed, along with mechanisms for carrying cache state across requests.

PersonTTS treats test-time scaling as a personalized controller-discovery problem rather than a one-dimensional accuracy-efficiency trade-off. It reuses experience across user profiles while evaluating each candidate against the target requirements.

DLoop lets a draft model run several confident drafting stages before the target model performs verification. The approach reduces target-model forward passes and reports 5% to 41% wall-clock speedup across several speculative decoding methods while preserving lossless decoding.

SlimWise separates optimization for prefill and decode in MoE serving: it runs prefill with the full model, then decodes with a pruned expert pool while reusing the original KV cache. The approach delivers up to 1.81× higher decode throughput at 50% expert pruning on Qwen3.6-35B-A3B.

RouteFM treats LLM routing as a reusable capability rather than a workload-specific fitting problem. By reading limited behavioral evidence from anonymous candidate models, a frozen router can adapt to new tasks, model pools, modalities, and context budgets.

PyTorch’s integration of Helion into vLLM’s linear backend explores whether a high-level kernel DSL can replace part of the manual specialization traditionally required for quantized GEMM. On NVIDIA Hopper, shape-specific tuning and hybrid dispatch produced end-to-end gains across the evaluated workloads, exceeding 10% in some cases.

MassAlloc Attention keeps access to every legal causal QK interaction, then uses normalized attention contribution to reduce post-score work in low-impact regions. The result is adaptive compute allocation with accuracy close to dense FullAttn.

Looped language models gain effective depth by repeatedly applying a shared block, but this also creates a sequential decoding bottleneck. Wavefront Decoding uses intermediate states as drafts and schedules states from different positions and recurrence depths in a diagonal, batched wavefront without additional training.

A KAIST AI study finds that longer self-reflection does not always expand a small reasoning model’s solution space. FlyBy lets the model diagnose its bottleneck and selectively query a stronger model when external knowledge is needed.

Disaggregated Quantization, or DQ, treats LLM prefill and token-by-token decode as separate systems problems. By specializing formats, weights, and placement for each phase, the approach targets both low-bit accuracy and time-to-first-token performance.

HybridInfer uses reinforcement learning to choose among on-device, edge, and cloud LLMs on real Android hardware. The work argues that sustained mobile inference can cause runtime instability, not merely slower generation.

RAZOR is a training-free method for pruning experts in mixture-of-experts models. Instead of relying only on usage or output magnitude, it estimates whether surviving experts can replace the function of a removed expert.

A new arXiv paper formalizes lossy speculative decoding and explains how allowing a controlled distribution shift can sometimes improve quality rather than merely reduce it. The work connects decoding to boosting and develops divergence-agnostic tools for optimizing the resulting distributions.

Open-source Colibrì treats NVMe storage as a slower tier for model weights, loading only the MoE experts needed for the current tokens. The approach lowers the memory barrier, but not the cost of data movement or the limits of inference speed.

Inferact reports that 16 TPU v7 Ironwood chips reached 709 tokens per second on Kimi K3, versus 452 tokens per second on 16 GB200 GPUs. The claimed advantage comes largely from software-level scheduling and memory optimization.