vLLM Brings Day-0 Serving Support for Kimi K3

Lead

As Moonshot AI made the Kimi K3 weights public, vLLM announced day-0 support for serving the model. The notable part is not merely that the engine can load a new open-weight model. vLLM is presenting a full serving path for a 2.8-trillion-parameter multimodal MoE with a one-million-token context window and a non-standard attention stack.

Key points

• A large hybrid MoE architecture: Kimi K3 activates 16 out of 896 experts per token. It combines Kimi Delta Attention, Attention Residuals, LatentMoE, native MXFP4 weights, and vision support, which means inference has to coordinate sparse expert routing, quantized weights, long-context memory, and multimodal input handling.
• Hybrid cache management is central: Most Kimi K3 layers use KDA, a recurrent linear-attention mechanism with compact state, while periodic full-attention layers retain exact global recall. vLLM therefore manages two memory types under one scheduler: paged KV blocks for full-attention layers and recurrent-state blocks for KDA layers.
• Prefix caching needed a redesign: Standard prefix caching assumes token-level KV entries. KDA updates recurrent and convolutional state at each token, but keeping a snapshot at every possible prefix boundary would be too expensive. vLLM decouples physical KDA state blocks from fine-grained prefix matching, registers snapshots within those blocks, and copies reusable state before extension. This allows long shared prompts to reuse both KDA state and paged KV.
• Speculative decoding is supported from launch: vLLM integrates Inferact’s open-sourced DSpark speculator for Kimi K3. According to the source material, on 16 NVIDIA GB300 NVL72 GPUs, throughput rises from 118 tokens per second without speculative decoding to 370 tokens per second with DSpark, a 3.14x improvement.
• Production-oriented features are included: The release covers prefill/decode disaggregation, agentic KV caching with Mooncake, tool calling, reasoning output, and structured output. It also targets NVIDIA Hopper and Blackwell platforms as well as AMD MI355X. Because of complex dependencies, the current path relies on Docker images, including pre-release components such as FlashInfer.

Why it matters

Kimi K3 stresses the inference stack in several ways at once. Its serving path has to handle sparse MoE execution, long-context state reuse, recurrent attention, full-attention layers, quantized expert weights, and speculative decoding. If any of these pieces is treated as an afterthought, the model may be technically runnable but impractical at scale.

That is why vLLM’s day-0 support is significant: it shows that modern inference engines increasingly need model-aware systems work, not just generic transformer kernels. The hybrid prefix caching work is especially important because it extends caching beyond standard KV-only models to architectures that keep recurrent state. vLLM says this machinery should benefit other hybrid linear models as well.

For developers, the near-term benefit is a clearer route to experimenting with Kimi K3 through documented serving commands and containerized environments. For the broader ecosystem, the release points to a larger trend: as open-weight models adopt longer contexts, sparse experts, and custom attention mechanisms, inference frameworks will compete on how quickly they can turn new architectures into efficient, production-ready services.

Source: vLLM Blog