How vLLM Serves GLM-5.3 with Million-Token Contexts on 16 H200s

Introduction

Agentic workloads accumulate tool results, file contents, and previous responses over multiple turns. For GLM-5.3, that growth exposes two constraints on H200 systems: the model weights leave limited room for KV cache, while increasingly long prompts make prefill a system-wide bottleneck. vLLM’s latest deployment experiment addresses both issues by placing prefill and decode on separate 8× H200 nodes and combining several forms of parallelism with memory offload.

The core design

• PCP splits queries by token. Prefill Context Parallelism assigns different portions of a prompt to eight GPUs. Each GPU computes all 64 attention heads for its assigned tokens, rather than receiving only a subset of heads as in tensor parallelism. Because causal attention makes later tokens more expensive, the prompt is divided into 16 chunks and front and back chunks are paired to balance work. The indexer is also computed only for local tokens instead of being replicated on every rank.
• DCP shards persistent KV storage. PCP improves prefill compute, but by itself leaves every GPU with a complete KV cache. Decode Context Parallelism distributes cache tokens round-robin, so each rank stores one eighth of the persistent cache. The example in the source puts a 1M-token request at 53.4 GiB of FP8 KV, or about 6.7 GiB per GPU after sharding.
• The memory gain requires communication. Since the query layout and KV layout do not match, the prefill engine gathers the distributed cache temporarily at every layer, performs local attention, and then releases the copy. The reported overhead is about 6% per step at shorter cached contexts and rises to roughly 15% near 248K tokens. DCP therefore trades interconnect traffic for a much larger usable KV pool.
• Hybrid HiSparse handles decode pressure. The decode node uses eight data-parallel ranks and keeps part of the KV state in host memory. The prefill node also uses CPU KV offload so later turns can reuse prefixes without keeping everything in HBM.

Why conventional TP and DP fall short

With data parallelism, a long prompt is assigned to one GPU, so the other seven cannot accelerate its prefill. The request’s cache must also fit on that card. Tensor parallelism distributes attention heads, but MLA’s latent KV is shared across all heads, meaning the full cache is replicated. Its indexer is likewise repeated, even though it scores earlier tokens before sparse attention selects a fixed set. PCP distributes the prompt’s query work, while DCP removes the cache replication; the two techniques target different limitations.

Significance

The experiment illustrates that million-token serving is not solved by adding a single parallelism mode. Compute placement, cache layout, host-memory capacity, and interconnect traffic must be designed together. vLLM reports that PCP8×DCP8 prefill with Hybrid HiSparse decode achieved 1.7–3.2× the per-user speed of a data-parallel deployment at equal throughput on AgentX. It was also the only tested configuration that served 1M-token contexts.

The result has clear trade-offs. DCP repeatedly moves a full cached context through the interconnect, although overlapping communication with computation may reduce that cost. Host offload shifts pressure from HBM to system memory and the transfer path rather than eliminating it. Still, for agents with long histories and repeated tool calls, the architecture offers a practical division of labor: PCP accelerates long prompts, DCP expands persistent cache capacity, and Hybrid HiSparse extends decode concurrency beyond GPU memory.

Source: vLLM Blog