Speculative Decoding in vLLM on AMD GPUs

Introduction

Autoregressive generation advances one token at a time. After each token is produced, it is appended to the sequence and becomes input for the next decode step. This simple process is reliable, but long generations require many sequential target-model steps, making decoding a major constraint for latency and serving throughput. A recent vLLM guide examines speculative decoding in AMD Instinct MI300X and MI355X environments running on ROCm, with an emphasis on practical configuration and tuning.

The draft-and-verify loop

Speculative decoding does not replace the original model. The original model remains the target and still determines the final output behavior. A lightweight draft component first proposes several future tokens. The target model then evaluates the proposed sequence in a verification pass, checking positions from left to right.

Accepted candidates can be committed together. At the first rejection, later candidates from that proposal are discarded, and the target model supplies the token that determines how generation continues. The benefit comes when several candidates survive: one target-side verification step can advance the sequence by multiple output tokens instead of only one.

Four practical families in vLLM

The guide describes three architectural groups and several representative methods:

• Native MTP is built into the target-model architecture and uses an auxiliary prediction path to generate candidates sequentially.
• Separate MTP drafters use a paired checkpoint, target-model activations, and shared KV-cache information for sequential proposals.
• Target-conditioned draft networks include EAGLE-3, DFlash, and DSpark. EAGLE-3 drafts autoregressively from target hidden states, DFlash proposes blocks in parallel, and DSpark adds lightweight causal correction and confidence-based prefix selection.

These labels describe the draft component rather than the target-model family. A target model may support native MTP while also being paired with separately trained EAGLE-3, DFlash, or DSpark models. Depending on the method, the drafter may consume a target hidden representation, states from selected layers, KV-cache data, or combined target-side features.

Why benchmarking must be workload-specific

Speculative decoding is not automatically faster simply because it proposes more tokens. Longer proposals can increase draft and verification costs, while rejected candidates reduce the number of useful tokens committed in each round. The guide reports that output-token throughput varies with the drafting method, proposal length, model family, draft checkpoint, workload, and acceptance behavior.

For deployment, this makes acceptance observability as important as raw model speed. A sensible process is to establish an ordinary autoregressive baseline, enable one drafting method at a time, and compare throughput and latency under representative requests. Configuration should be tuned around the behavior of the paired target and draft models rather than copied as a universal recipe.

Impact

The broader lesson is that speculative decoding is a systems optimization, not merely a decoding trick. On AMD GPUs, vLLM exposes several draft designs with different trade-offs between proposal cost, parallelism, model compatibility, and candidate quality. Its value appears when the draft component is sufficiently aligned with the target model to let each verification pass commit more useful output. In practice, the key question is not how many tokens the system can guess, but how many guesses the target model can consistently accept.

Source: vLLM Blog