Inference & Serving

Researchers from UC Berkeley and MIT have open-sourced FreeToken, an inference engine designed to reduce the cost of moving sparse MoE weights between system memory and consumer GPUs. The paper reports about 39 tokens per second for Qwen3.6-35B on a laptop RTX 4060.

A study of a hybrid 27B language model challenges the assumption that recurrent Gated DeltaNet layers must remain at 8 or 16 bits. Quantizing all 496 linear layers with NVFP4 W4A4 stayed close to BF16 across language, reasoning, coding, and long-context retrieval tests.

Cerebras has listed Qwen 3.8 27B on its public inference endpoints, with access available to free-trial and pay-as-you-go users. The provider lists about 1,500 tokens per second and documents its context limits and compression policy.

Nvidia has released Personal AI Router, or PAIR, an open-source tool that connects compatible computers on a home network for local AI inference. Despite its name, it is software rather than a networking device.

CRISP redesigns input-adaptive sparse prefilling with a direct structural routing metric and a sink-aware threshold. The approach targets both the overhead of dynamic routing and the noise introduced by cumulative attention-mass selection.

Declarative Attention lets a language model signal which part of a long context it needs during generation. Early evaluations suggest substantial reductions in attended tokens, with a modest but nonzero accuracy cost.

A production-driven post-training recipe consolidates traffic from more than 200 internal applications onto one self-hosted model. It separates instruction following, tool use, and internal task distribution into specialized GRPO experts before merging them with two-stage SLERP.

NAVER AI Lab introduces Verification-Aware Training (VAT), a plug-in objective for speculative decoding that models sequential verification during draft training. The method improves accepted sequence length and measured inference speed without changing the model architecture or decoding procedure.

vLLM-Omni treats MiniMax H3 as an end-to-end serving problem, optimizing attention, diffusion operators, VAE decoding, transport, and MP4 assembly. FastVideo’s FastH3 then reduces the diffusion schedule to four DiT forwards and brings complete-response latency close to playback time.

A new comparison finds that sliding-window attention with sinks matches or outperforms post-trained linear attention across several models and tasks. Its strongest advantage appears in long-context reasoning, without requiring additional post-training.

A new study reformulates an LLM’s output projection and Top-k selection as maximum inner product search. An HNSW index retrieves only promising token candidates, targeting the memory-bandwidth bottleneck in small-batch CPU inference.

A long-context experiment suggests that inference backends, KV-cache precision, quantization, and multi-GPU communication can change token choices even when the model weights stay identical.

TileMix mixes FP16 and INT8 score computation inside a fused dense-attention kernel by routing hardware-aligned tile groups. The approach keeps all legal token interactions while targeting a better long-context prefill trade-off between quality and throughput.

Structural compression followed by 4-bit quantization can weaken reasoning, math, coding, and long-context performance. Multiverse Computing’s Quantization-Aware Healing trains the quantized student directly against the original, uncompressed model.

Multiverse Computing’s Quantization-Aware Healing method distills a structurally compressed, 4-bit model directly from the original full-size teacher. In a GPT-OSS experiment, the resulting 60B MXFP4 model outperformed its 60B BF16 counterpart on seven of nine benchmarks.

Daedalus-150M was designed from its deployment target backward: single-user, one-token-at-a-time decoding with 4-bit weights on a conventional CPU. By replacing two-thirds of its attention blocks with cache-bounded short convolutions, the model targets the memory costs that become dominant at longer contexts.

ParaTempo is a training-free asynchronous framework that manages parallel reasoning branches by tracking how their tentative answer distributions converge over time. It prunes unstable paths, retires persistent ones early, and reallocates computation to new branches.

Nexus separates MCP tool selection from the costly repeated prefill of full tool schemas. Its retrieval path scales more reliably, while KV-cache splicing delivers bounded gains because rotary position drift limits safe placement.

Llama-Mobile introduces a quantization pipeline for deploying vision-language models on resource-constrained devices. It uses model-generated data and an Arm-oriented 2.7-bit format to compress Llama 3.2 11B Vision Instruct to 3.7 GB while retaining strong results on standard visual question answering tasks.

vLLM’s practical guide explains how speculative decoding works on AMD Instinct GPUs with ROCm. It compares native MTP, EAGLE-3, DFlash, and DSpark while stressing that real gains depend on acceptance behavior and workload.

The same model weights can produce different text on different GPUs, runtimes, quantization settings, and attention kernels. The apparent loss of intelligence may come from the inference stack rather than the model itself.