DLite Removes History from Speculative Decoding for Agentic AI

Why agentic workloads expose a weakness in speculative decoding

Speculative decoding accelerates language-model inference by splitting generation into two stages. A small draft model proposes several tokens, and the target model verifies them together. When many proposed tokens are accepted, the target model performs fewer sequential decoding steps, reducing latency without changing the verified output distribution.

The approach becomes harder to sustain in agentic systems. An agent may repeatedly call tools, receive external observations, revise its plan, and continue the conversation. Each turn adds information to the context, while the relevant state keeps changing. According to the paper, existing draft models can see their acceptance length fall by as much as 80% under these workloads. Keeping a full historical KV cache for the drafter also adds memory and systems overhead.

Reusing information the target has already computed

DLite treats the growing history as the common cause of both problems. Rather than asking the drafter to preserve and repeatedly model the complete interaction history, it reuses contextual hidden states produced by the target model during inference. The drafter then makes local speculative predictions from those representations without explicitly maintaining the historical context on its own side.

This is the central meaning of “history-free” in DLite. The method does not claim that history is irrelevant to generation. Instead, historical information is consumed through representations already created by the target model, rather than being duplicated as a separate drafter-side sequence and KV cache. The prediction problem is consequently reframed from modeling an ever-growing conversation to making a simpler local prediction from contextual features.

A second component is a lightweight sequential head. Candidate tokens in a speculative block are often generated in parallel, but later positions should still depend on earlier positions. The sequential head restores these intra-block causal dependencies with limited additional cost. It is intended to preserve the speed benefit of block prediction while reducing the mismatch caused by treating all positions as independent.

Training and reported results

DLite combines target-model hidden states with a local prediction module and a sequential head, making it different from a conventional standalone draft model. The authors therefore introduce a two-stage training strategy tailored to this heterogeneous architecture. The supplied material does not provide the full training recipe, but describes the strategy as a way to align history-free prediction with the causal structure inside each speculative block.

Experiments cover general-purpose, long-context, and agentic workloads. The paper reports acceptance lengths of 2–6 and states that DLite outperforms the state-of-the-art speculative-decoding method DSpark by up to 70%. These findings suggest that eliminating duplicated historical state can affect more than memory usage: it may also make draft prediction more stable as the context evolves.

What the result means for inference systems

DLite’s broader contribution is a change in where context should live during speculative decoding. For an agent, the drafter may not need to maintain a second full representation of the entire interaction. Reusing target-side hidden states could reduce cache pressure in long sessions and make the drafter’s cost less sensitive to conversation history.

The abstract alone does not establish how the method behaves across hardware platforms, model sizes, or deployment schedules. Acceptance length is also not identical to end-to-end throughput: hidden-state movement, target verification, batching, concurrency, and memory management will all influence production performance. Even so, DLite offers a focused design direction for agentic inference—avoid duplicating an expanding history, and build local speculation on information the target model has already encoded.

Source: Hugging Face Daily Papers