Inference & Serving

A long-context experiment suggests that inference backends, KV-cache precision, quantization, and multi-GPU communication can change token choices even when the model weights stay identical.

TileMix mixes FP16 and INT8 score computation inside a fused dense-attention kernel by routing hardware-aligned tile groups. The approach keeps all legal token interactions while targeting a better long-context prefill trade-off between quality and throughput.

Structural compression followed by 4-bit quantization can weaken reasoning, math, coding, and long-context performance. Multiverse Computing’s Quantization-Aware Healing trains the quantized student directly against the original, uncompressed model.

Multiverse Computing’s Quantization-Aware Healing method distills a structurally compressed, 4-bit model directly from the original full-size teacher. In a GPT-OSS experiment, the resulting 60B MXFP4 model outperformed its 60B BF16 counterpart on seven of nine benchmarks.

ParaTempo is a training-free asynchronous framework that manages parallel reasoning branches by tracking how their tentative answer distributions converge over time. It prunes unstable paths, retires persistent ones early, and reallocates computation to new branches.

Daedalus-150M was designed from its deployment target backward: single-user, one-token-at-a-time decoding with 4-bit weights on a conventional CPU. By replacing two-thirds of its attention blocks with cache-bounded short convolutions, the model targets the memory costs that become dominant at longer contexts.

Nexus separates MCP tool selection from the costly repeated prefill of full tool schemas. Its retrieval path scales more reliably, while KV-cache splicing delivers bounded gains because rotary position drift limits safe placement.

Llama-Mobile introduces a quantization pipeline for deploying vision-language models on resource-constrained devices. It uses model-generated data and an Arm-oriented 2.7-bit format to compress Llama 3.2 11B Vision Instruct to 3.7 GB while retaining strong results on standard visual question answering tasks.

vLLM’s practical guide explains how speculative decoding works on AMD Instinct GPUs with ROCm. It compares native MTP, EAGLE-3, DFlash, and DSpark while stressing that real gains depend on acceptance behavior and workload.

The same model weights can produce different text on different GPUs, runtimes, quantization settings, and attention kernels. The apparent loss of intelligence may come from the inference stack rather than the model itself.

FlashPrefill V2 upgrades sparse attention for long-context LLM prefill with mean correction, GPU-oriented kernel engineering, and native support for paged KV caches and continuous batching. On NVIDIA H20, it reports up to 47.26× the speed of FlashAttention-2 at 128K context in FP8.

Ramp has productized Router, an AI model routing service that gives companies a single API for accessing and switching among multiple language models. It is currently available only in the United States and will waive its routing fee for the rest of 2026.

FreeToken is an edge-native serving system designed to run large mixture-of-experts models across heterogeneous personal hardware. Instead of treating a PC as an undersized GPU, it dynamically coordinates CPU, GPU, memory, storage, and model state.

The DumpsterCluster project shows that retired GPUs can be assembled into a practical large-model inference platform. Its low hardware cost comes with a major condition: electricity prices and grid carbon intensity determine whether the system is truly economical or sustainable.

A security-focused investigation describes how unused AI credits are being bought from startups and resold through brokers, marketplaces, and discount routing services. The emerging trade turns inference capacity into a quasi-currency with cost, compliance, and abuse implications.

LiveAnimate targets a familiar tradeoff in pose-driven human animation: diffusion models can look good, but they are too slow and too memory-hungry for interactive use. The paper reframes a 14B video DiT into a streaming system built for stability over long sequences.

UniMoMo is a post-training compression method for large recommendation MoE models. It merges experts by functional behavior on calibration data and protects high-traffic routing paths to preserve quality while improving serving speed.

vLLM’s Decode Context Parallelism shards KV cache along the sequence dimension rather than relying only on tensor parallelism across attention heads. The approach is aimed at long-context agentic workloads where cache memory, not raw compute, often limits throughput.

GPTQ-2D addresses a more general form of adaptive matrix rounding where basis matrices act on both sides of the residual. Instead of vectorizing the problem and paying quartic cost, it preserves the same rounded output with a cubic-time anti-diagonal schedule.

OmniScope argues that audio and video should not blindly guide each other during token compression. By using the query as a shared anchor while estimating salience per modality, it improves the efficiency-accuracy trade-off for omnimodal LLM inference.

The paper examines lossy verification in speculative decoding and shows why relaxed verification can quietly change the target model’s decoding distribution. Its core message is that speedups may come with unstable or degraded generation quality.