Evolving Infra: Adaptive LLM Serving for Agent Workloads

Introduction

LLM serving for agentic applications is more complicated than serving isolated prompt-and-response queries. A coding agent, a mathematical reasoning agent, or a computer-use system may invoke a model repeatedly, extend its context over several steps, and alternate between phases with very different compute characteristics. The serving layer must therefore balance Prefill work, ongoing Decode work, and the storage and movement of reusable context.

The paper Evolving Infra: Agent-Adaptive LLM Serving at Scale, featured by Hugging Face Daily Papers, presents a serving architecture built around that observation. Instead of treating Prefill and Decode capacity as a permanently fixed partition, it adapts the allocation at runtime. It also uses signals from agent context management to make KV-cache eviction and transfer decisions.

Key ideas

• Separate compute roles from cache management. Existing Prefill-Decode (PD) disaggregated systems commonly assign nodes to one role or the other with limited flexibility. Evolving Infra decouples compute allocation from KV-cache management, allowing nodes to switch between Prefill and Decode more efficiently.
• Plan the PD split online. An online planner matches the Prefill-Decode division to the current composition of the workload. The goal is to reduce situations in which one pool is queued while the other has unused capacity.
• Make cache decisions agent-aware. Agents often return to context accumulated in earlier steps. The proposed distributed KV manager uses signals related to agent context management to guide cache eviction and movement across nodes, rather than relying only on coarse serving signals.
• Evaluate across tasks and hardware scales. The study examines coding, math, and computer-use workloads with GLM-5.2 and Kimi K2.6, using as many as 256 GB300 GPUs. The authors report up to a 4.1x reduction in mean end-to-end request latency and up to a 14x reduction in P95 time to first token compared with NVIDIA’s Dynamo production framework.

Why it matters

The broader contribution is a shift from static resource orchestration toward serving systems that respond to workload behavior. For agents, latency is influenced not only by the model’s token computation, but also by repeated calls, changing phase mixes, and failures to preserve context that could have been reused. A scheduler that accounts for request phase, workload composition, and context lifetime may be better positioned to balance compute utilization with cache reuse.

The reported gains should still be interpreted within the paper’s stated setup: specific tasks, models, hardware, and a comparison with Dynamo. They do not automatically establish the same improvement for every production deployment. Operators would also need to consider planning overhead, cross-node cache-transfer costs, and whether their agent framework exposes useful context signals.

Even with those qualifications, Evolving Infra points to an important direction for LLM infrastructure. As agentic workloads become a larger share of inference demand, serving systems may optimize for more than token throughput. They may increasingly adapt to how agents call models, accumulate context, and reuse intermediate state.

Source: Hugging Face Daily Papers