Why Cancellation Alone Cannot Save an Overloaded LLM Engine

Introduction

When an LLM request times out, clients and proxies often retry it automatically. The usual assumption is that propagating cancellation will free the engine to recover. A new arXiv study using vLLM challenges that assumption: by the time cancellation reaches the inference engine, a retry may already have consumed most of the work needed to process its prompt.

What the experiments found

The researchers tested overload scenarios with three open models on vLLM. For one model, they also collected scheduler traces that connected each client attempt with the execution created inside the engine. The results expose several layers of waste:

• Cancellation alone did not restore service. With retries enabled, no run recovered through cancellation by itself. Only 12% to 38% of calls arriving after the overload were ultimately delivered.
• The cancellation window is extremely narrow. Retries that vLLM actually ran typically began prefilling less than half a second before timing out. Executions later aborted by vLLM had already processed 79% to 85% of their prompt tokens.
• A common counter can understate the damage. vLLM’s prompt-token counter missed roughly one-third to one-half of all prompt tokens that were computed. A dashboard based only on that metric could therefore underestimate wasted work and the cost of retries.

The study also compared two alternatives. A retry budget restored service by limiting additional attempts, but many calls from the overload period remained undelivered. Request coalescing combined duplicate attempts and delivered every call from the overload onward when cancellation had not already aborted the execution in progress. Some of those responses arrived late, but they were not discarded merely because the clients had retried.

Why it matters

The paper suggests that “cancellation succeeded” is an incomplete operational metric. A request can be marked cancelled at the API layer while the engine has already spent most of its compute on it. Conversely, a system can appear to have recovered while users still have not received the calls that triggered the overload.

For LLM operators, retry policy must be treated as part of capacity management rather than as a purely client-side reliability feature. Useful controls may include bounded retries, early duplicate detection, request coalescing, and accounting that captures all prompt tokens actually computed. The study does not establish one universal solution, but it does provide a practical evaluation rule: measure both the executions retries add and the calls clients ultimately receive.

Source: arXiv