How vLLM Is Reworking Inference Serving for Agentic Workloads

Introduction: the serving target is changing

Many inference systems are designed around a single request, a bounded context, and a relatively long generation. Agentic applications have a different rhythm. An agent may call tools, inspect results, delegate work to subagents, and return to the same conversation dozens of times. Each turn appends new information to the accumulated context. The input grows steadily, while the newly generated answer is often short.

That pattern changes the optimization problem. A serving system must not only generate tokens quickly; it must preserve and retrieve large KV states, keep reused prefixes close to computation, and balance progress latency against hardware utilization. In a vLLM post based on SemiAnalysis AgentX, the project outlines a coordinated approach to these requirements.

What AgentX reveals

• Long sessions and asymmetric token volumes. The benchmark reports a median of 43 turns per session, a median input of about 142K tokens, and a median output of 444 tokens. The workload repeatedly submits an expanding context, but much of that context has already been processed.
• Heavy prefix reuse. Prefix-cache hit rates are above 96%. This reduces the need to recompute the same prefix, but creates pressure to retain and move KV data efficiently across GPUs, workers, and storage tiers.
• Subagent fan-out. Forty-four percent of sessions include at least one subagent, and those sessions have a median of four subagent rollouts. Forking and joining context complicates cache affinity and routing decisions.
• A two-dimensional service objective. Cost determines how many agents can run on a fixed hardware budget, while interactivity determines how quickly an agent completes its next reasoning or tool-use step. Optimizing only aggregate throughput can miss the user-visible bottleneck.

Three layers of vLLM’s approach

At the data layer, vLLM extends its PagedAttention heritage with a hybrid KV-cache manager. Modern models may combine full attention with sliding-window or linear attention, whose cached states have different sizes and lifetimes. Rather than statically partitioning memory by attention type, the design uses a uniform page as the allocation unit and a shared block pool. Capacity can therefore be redistributed as concurrency, sequence length, and reuse patterns change.

The post also describes a packed KV layout for DeepSeek V4. An earlier layout spread cache types across many tensors and size buckets, which introduced padding and transfer overhead. The packed arrangement places cache groups and layers in a contiguous backing allocation for each block. According to the post, this reduces descriptor and Prefill/Decode transfer overhead; when the FP4 indexer is enabled, it can save roughly 10% of KV-cache memory.

For capacity beyond GPU memory, vLLM integrates Mooncake Store as a distributed KV-cache pool. Its hierarchical design can include CPU memory and disks, while standalone-store mode lets external clients manage those tiers and leaves vLLM workers as requesters. Integrations with routers such as Dynamo and llm-d are intended to make cache hits less dependent on a request remaining on one particular instance.

At the execution layer, the work covers model-specific parallelism, kernels, asynchronous scheduling, and speculative decoding. Hybrid attention also makes CPU-side lookup more expensive, so vLLM reports improvements through more efficient data structures, asynchronous lookups, work moved away from the scheduler’s critical path, and parallel send/receive operations.

The control layer focuses on Prefill/Decode disaggregation. There is no universally optimal ratio: context lengths, hit rates, subagent behavior, and concurrency can vary widely. Routing must therefore balance cache locality with load distribution, and the best ratio may shift as traffic changes.

Why the results matter

For selected AgentX configurations, vLLM reports up to 130K total tokens per GPU-second on DeepSeek V4 Pro and up to 376 tokens per second of interactivity on MiniMax M3. Across DeepSeek V4 Pro, MiniMax M3, and Kimi K3, the post reports a 14.6x to 106x serving-cost advantage relative to Opus 5 API pricing. These are benchmark observations tied to specific models, hardware, and configurations, not universal deployment guarantees.

The broader lesson is that agentic inference is becoming a context-infrastructure problem. Cache placement, cross-worker transfers, subagent state sharing, and dynamic Prefill/Decode balancing can be as important as the raw decoding kernel. Operators evaluating such systems will likely need metrics that include long-session latency, cache reuse, transfer overhead, concurrency behavior, and total cost of ownership—not just tokens per second.

Source: vLLM Blog