vLLM Adds Day-0 Support for TML Inkling, Bringing a 1T Multimodal Model to Optimized Serving

Lead

vLLM has announced day-0 support for TML Inkling, a new 1T-parameter multimodal model from Thinking Machines Lab. The update is notable not simply because another model can now run in vLLM, but because Inkling combines several serving challenges at once: multimodal inputs, very long context, Mixture-of-Experts layers, relative attention, short convolution, and multi-token prediction for speculative decoding.

The integration covers both thinkingmachines/Inkling-NVFP4 and the BF16 thinkingmachines/Inkling model. According to vLLM, the implementation currently targets NVIDIA Blackwell and Hopper GPUs, with broader hardware support still in progress.

Key points

• A large multimodal model: Inkling accepts text, image, and audio inputs and generates text. Its decoder-only Transformer backbone supports up to 1M tokens of native context.
• Long-context efficiency: The model uses 66 layers, including 11 full-attention layers and 55 sliding-window attention layers. This heavy reliance on sliding-window attention is central to making million-token context practical.
• Non-standard attention design: Instead of RoPE, Inkling uses relative attention, adding a learned relative-position term to attention logits.
• Short convolution throughout the stack: Each layer applies short convolution with window size 4 to attention keys, attention values, attention output, and MoE output. vLLM treats the sconv state like the KV cache of a virtual sliding-window attention layer, allowing unified cache management and prefix caching.
• MoE with expert sinks: Each layer has 256 routed experts with top-6 routing plus two shared experts. These shared experts participate in routing-score computation but are excluded from top-6 selection.
• MTP as a serving accelerator: Inkling includes eight MTP heads for speculative decoding, potentially allowing up to nine tokens per forward step. vLLM recomputes MTP KV cache when draft tokens are rejected, avoiding stale-cache issues.

Performance and validation

vLLM reports up to 380 tokens per second per user with MTP8, with a mean acceptance length of 4.5, on four NVIDIA GB200 GPUs. Without MTP, the reported figure is 140 tokens per second per user. The benchmark used 8K-token prompts sampled from SPEED-Bench and generated 1K output tokens per request.

The implementation was checked across multiple benchmarks: MMAU for audio, MMMU-Pro for vision, BFCL for tool calling, HLE for reasoning, and NIAH for long context. vLLM says its results match the reference implementation across modalities and capabilities; for long context, it reports exact agreement through 221K tokens and roughly within one percentage point through 513K, while very extreme context lengths show higher run-to-run variance.

Why it matters

This release highlights a broader shift in inference infrastructure. Modern model serving is no longer just about loading weights and producing tokens; it requires co-optimizing cache layout, communication patterns, parallelism, speculative decoding, and specialized kernels around each model architecture.

For developers, the value is operational: vLLM aims to hide much of the complexity behind an inference-serving interface while preserving advanced features such as LoRA, TP/DP/EP/PP parallelism, prefix caching, and disaggregated serving. For model builders, day-0 support signals tighter coordination between frontier model releases and the serving stacks needed to deploy them.

Source: vLLM Blog