How vLLM Tunes PD Serving for Qwen3.8-2.4T

Introduction

Qwen3.8-2.4T presents an unusually demanding serving problem: it combines a large MoE model with a hybrid sequence architecture. vLLM’s latest study evaluates disaggregated Prefill-Decode, or PD, serving on a GB300 NVL72 cluster under an 8K input and 1K output workload. Rather than optimizing only for maximum throughput, the study maps both ends of the throughput-latency trade-off.

The reported throughput-oriented result is about 5,000 total tokens per GPU. In a configuration aimed at interactive performance, the system reaches 180 generated tokens per user. These figures should be read as points on a measured Pareto frontier, not as a single universally optimal configuration.

Key points

• KV-cache capacity sets the concurrency ceiling. The model has 92 layers: 69 GDN layers and 23 Full-Attention layers, with an MoE block in every layer. Full-Attention state grows with the number of tokens, while GDN state is maintained per request. That difference materially changes memory planning.
• GDN state determines the block granularity. The post estimates about 4.125 MiB of GDN state per cache block and a block capacity of 2,112 tokens. For an 8K/1K request, Full-Attention and GDN layers consume blocks differently, and their combined footprint determines request-level cache usage.
• Weight memory is only one part of the budget. Driver overhead, the CUDA context, NCCL buffers, allocator space, peak activations, and CUDA Graph allocations all reduce the memory available for KV cache. The gpu_memory_utilization setting also reserves headroom for effects that cannot be predicted perfectly in advance.
• Prefill and Decode must agree on transfer layout. The two engines can estimate their block sizes independently, but KV-cache transfer requires compatible sizes. If they differ, operators may need to set --block-size manually, potentially wasting some cache capacity.
• Topology must follow the target metric. The reported experiments compare TP8 and TP4DP4 arrangements with different sequence and batch limits. Raising concurrency alone is not enough; it must fit within activation, graph, and cache constraints.

Why it matters

The most reusable part of the study is its troubleshooting sequence. Start by estimating state per request, then subtract weights and runtime reservations from device memory, measure activation peaks, and finally sweep concurrency, batch limits, and parallel layouts to construct a throughput-latency frontier. This approach is especially relevant for models that mix attention, recurrent state, and MoE layers, where parameter count alone is a poor guide to serving capacity.

The work also highlights that PD serving is not simply a way to separate compute stages. Cache layout, transfer compatibility, and independent resource planning for Prefill and Decode can determine whether a theoretically attractive configuration works in practice. By documenting the reasoning behind its recipes, vLLM provides a framework for adapting the process to other models instead of treating one benchmark configuration as a universal answer.

Source: vLLM Blog