Following the Bottleneck: How vLLM Optimized MiniMax M3 on AMD MI355X

Introduction

Optimizing an MoE model is rarely a matter of finding one universally fastest kernel. vLLM’s review of MiniMax M3 on AMD Instinct MI355X presents a more useful workflow: inspect the shape that actually reaches each rank, remove work repeated across layers or tokens, verify which bytes move, confirm that the intended fast path is being dispatched, and then move up to the queue when leaf kernels stop dominating runtime.

The gains came from a sequence of changes

Under the fixed-topology MXFP8 standard-serving configuration, output throughput at concurrency 32 rose from 109.1 to 342.4 tokens/s/GPU, a 3.14× improvement. Median TTFT fell from 1.46 to 0.67 seconds, while mean TPOT dropped from 69.1 to 22.1 milliseconds. At concurrency 128, the same TP4/EP1 four-GPU path increased from 297.8 to 623.7 output tokens/s/GPU. MXFP4 first moved from 212.1 to 716.8 at concurrency 128 on TP4/EP1, and a later TP2/EP1 result reached 943.5. That final comparison also reflects higher deployment density and should not be read as a fixed-topology speedup. EAGLE3 reached 682.4 output tokens/s/GPU at concurrency 128 on TP4/EP1. With P/D disaggregation and a retuned topology, the system reached 6,370.5 total tokens/s/GPU at concurrency 512, with 1.32 seconds median TTFT.

Five lessons from the optimization path

1. Tune for local shapes, not model diagrams. Tensor parallelism, replicated KV and index heads, padding, and routing determine the actual local M, N, and K. At TP8, the Q heads are sharded, but the four KV and four index heads are replicated, so the fused QKV projection sees local N=1536 rather than simply the global dimension divided by eight. Separating large-M and small-M launch regimes improved TP8 8K/1K throughput by 7.8%–9.4%. Shape-aware tile selection then exposed more decode parallelism while reducing loop iterations where appropriate.

2. Fuse work that is mathematically required but operationally separate. The shared expert originally ran as an independent dense MLP, creating extra launches, intermediate storage, and traffic. Appending it to the routed-expert table allowed grouped GEMMs to process both paths together without changing the model’s required computation. The improvement was 30.2% at concurrency 1 and 5.6% at concurrency 128, a pattern consistent with launch overhead being more visible at low load.

3. Move invariants out of the serving loop. MXFP8 weight and scale reshuffling was moved to model load time. For speculative decoding, the MSA indexer changed from one workgroup per speculative token to one workgroup per request, processing draft positions together and reusing key loads. The index kernel improved by as much as 48.9%, while end-to-end serving improved by about 3.3% in the PR tests.

4. Count control-plane bytes too. Sparse attention reduces arithmetic but introduces score selection, block mapping, and page metadata. Because adjacent sparse layers often select similar blocks, later layers can reuse an earlier index decision. Mean TPOT fell by about 10% at concurrency 1 and about 4% at higher concurrency in the reported measurements.

5. When kernels flatten, follow the queue. Graph execution reduces launch overhead, but backend choice still depends on input length and concurrency. Native MXFP8 linear, emulated linear, and sparse paged attention each win in different regions. P/D disaggregation similarly requires validating KV handoff first, then adding capacity where requests actually wait.

Why it matters

The broader lesson is a portable performance model for sparse and MoE serving. Labels such as “prefill,” “decode,” or “TP4” are not detailed enough to select an implementation. A kernel-level gain also cannot be treated as an end-to-end gain: a 48.9% index-kernel improvement translated to roughly 3.3% in serving because other stages remained on the critical path. The most reusable recipe is therefore a loop—measure local shapes, remove repeated work, verify data movement and dispatch, and keep following the queue as the bottleneck moves.

Source: vLLM Blog