How vLLM Optimized Kimi K3 with DSpark on GB300

Introduction

Speculative decoding is useful only when the target model can accept enough of the draft model’s predictions in one verification pass. vLLM’s latest work brings its Speculators training library to Kimi K3 and demonstrates a DSpark deployment on a multi-node GB300 NVL72 system. The result is less a new serving API than an example of how speculative decoding can be turned into a repeatable training and deployment workflow.

What DSpark changes

Conventional speculative decoding uses a small model to generate candidate tokens autoregressively, after which the large target model verifies them in a batch. EAGLE-3 is a strong example of this design, but producing a longer candidate sequence still requires multiple sequential draft steps. DFlash reduces that cost by predicting an entire block in one non-causal backbone pass.

The tradeoff is coherence. Because positions in the block are predicted in parallel, later positions cannot fully condition on earlier sampled tokens. A wrong choice near the beginning can invalidate the remaining suffix, a failure mode described as suffix decay.

DSpark keeps the parallel backbone and adds several lightweight components:

• A Markov logit-bias head adjusts later token probabilities based on previously selected tokens.
• A confidence head estimates which predictions are likely to pass target-model verification.
• A hardware-aware scheduler verifies longer prefixes when the system is lightly loaded and trims low-confidence suffixes under pressure.

This design aims to preserve the low draft cost of DFlash while recovering some of the local dependency modeling associated with autoregressive drafting. The source reports longer accepted sequences than both DFlash and EAGLE-3 on Qwen3 targets, but those figures should not be interpreted as universal speedups across every model or workload.

Kimi K3 results

The released Kimi K3 speculator uses a five-layer draft model with roughly 5 billion parameters. It proposes eight tokens at each decoding step. Across nine evaluation domains, the macro-average acceptance length was 4.11 tokens per verification round. Mathematical reasoning was the strongest area at 6.42 tokens, followed by HumanEval at 4.96 and translation at 4.65.

Long context was another notable result. On LongBench-v2 with a 378K-token prompt, the system reached as many as 5.31 output tokens per decoding iteration. In a concurrency test, increasing the number of requests from one to 16 raised aggregate output throughput from 177 to 683 tokens per second. Median time to first token rose from 379 to 479 milliseconds, a relatively limited increase for that change in concurrency.

For mathematical reasoning, the article reports single-stream interactive speed rising from roughly 110 to 435 tokens per second. At matched interactivity, aggregate output throughput was reported to increase by as much as 3.5 times. These numbers remain dependent on batching, context length, target-model behavior, and the specific hardware and serving configuration.

Why the engineering matters

The broader contribution is the workflow. Speculators packages draft models in a Hugging Face-compatible format that vLLM can load directly, reducing the need for model-specific serving code. The deployment recipe combines multi-node tensor parallelism, FP8 KV cache, prefix caching, and an MLA-oriented attention backend, showing how algorithmic gains depend on the surrounding systems stack.

The GB300 NVL72 setup also highlights the importance of rack-scale testing. For practitioners, DSpark suggests that speculative decoding should be evaluated as a system rather than as an isolated model component. Acceptance length, time to first token, concurrent throughput, communication overhead, and long-context behavior are all necessary to understand its real-world value.

Source: vLLM Blog