Inference & Serving

The vLLM team and ecosystem partners have improved the Arm CPU serving path across packaging, allocator behavior, synchronization, dense layers, and paged attention. The result is broader model support and stronger inference performance on Arm Neoverse-based servers.

Ai2’s OlmoEarth Platform is designed to bridge the gap between open geospatial foundation models and the infrastructure needed to run them at continental scale.

vLLM and Speculators now support P-EAGLE, DFlash, and DSpark, three parallel drafting approaches designed to generate candidate token blocks in a single step. The update targets a key bottleneck in speculative decoding: the sequential cost of producing draft tokens.

vLLM has added day-0 serving support for Kimi K3, combining hybrid KDA prefix caching, DSpark speculative decoding, disaggregated deployment, and GPU-specific kernels. The release is less about simply loading a model and more about making a 2.8T-parameter hybrid MoE usable in production.

Runway has launched Media Router, a developer tool that automatically selects image, video, or audio generation models based on priorities such as quality, speed, and cost. The move shows how crowded generative media is pushing companies toward orchestration layers rather than single-model bets.

IBM Research argues that model routing in agentic systems is less a classification problem than a systems optimization problem. Real-world performance depends on cost, latency, caching, infrastructure, and governance constraints.

A new arXiv paper argues that production model optimization should start with deployment constraints, not with a menu of compression algorithms. It organizes decisions around data availability, latency, memory, accuracy tolerance, and retraining budget.

vLLM has added day-0 support for TML Inkling, a 1T-parameter multimodal model with native long-context capability, MTP speculative decoding, and optimized serving on NVIDIA GB200 GPUs.