Inference & Serving

IBM Research argues that model routing in agentic systems is less a classification problem than a systems optimization problem. Real-world performance depends on cost, latency, caching, infrastructure, and governance constraints.

A new arXiv paper argues that production model optimization should start with deployment constraints, not with a menu of compression algorithms. It organizes decisions around data availability, latency, memory, accuracy tolerance, and retraining budget.

vLLM has added day-0 support for TML Inkling, a 1T-parameter multimodal model with native long-context capability, MTP speculative decoding, and optimized serving on NVIDIA GB200 GPUs.