SpecPilot Uses Agents to Discover Better Speculative Decoding Systems

Why speculative decoding still leaves performance on the table

Speculative decoding asks a lightweight drafter to propose several tokens before a larger target model verifies them in parallel. The approach is attractive for memory-bound language-model generation because the target can accept multiple tokens during one forward pass. Yet the speedup available in theory is not necessarily delivered by a stock implementation.

For a single request, the target may still read its full set of weights while accepting only a small number of proposed tokens. Under batching, verification work is distributed across requests and can substantially reduce the benefit relative to autoregressive decoding. The best implementation also depends on the target model, drafter, GPU, and serving regime, making a universal optimization difficult.

An optimization loop for each deployment

SpecPilot addresses this problem with an autonomous discovery loop rather than a fixed decoding recipe. At the start of each round, a cost model estimates where the current system falls furthest short of its achievable ceiling. Multiple agents then explore different algorithmic and systems directions in parallel. Successful ideas are integrated and tested in later rounds.

The system has several safeguards and sources of leverage:

• Cost-guided exploration: Search focuses on the dominant cost in the selected deployment, such as weight movement, verification, or batching.
• Token forcing: Every output token is checked against the stock implementation, helping detect reward hacking in which a system appears faster by changing its outputs.
• Structured memory: Validated mechanisms are recorded and reused across rounds and deployment settings.
• Joint algorithm-system search: The discoveries cover not only implementation details but also verification structure, weight representation, and request organization.

The reported mechanisms include a general lossless 12-bit weight format, tree verification for hybrid recurrent targets, a two-level verification design, and three batching mechanisms. Together, they show that speculative decoding performance is shaped by the entire execution path, not just by how effectively the drafter predicts tokens.

Results and broader implications

The evaluation spans four target models, three drafters, two GPU platforms, and five benchmarks. For one request, SpecPilot increases speedup over autoregressive decoding from the stock system’s 4.46× to 10.87×. With batching, the reported improvement is from 1.94× to 3.93×. The authors state that these gains require neither additional training nor a quality trade-off.

The broader contribution is a change in how inference optimization is approached. Speculative decoding is unlikely to have one configuration that is optimal everywhere; its best design must reflect the model, hardware, drafter, and traffic pattern. SpecPilot turns that deployment-specific tuning process into a repeatable search procedure with explicit correctness checks. The reported numbers remain deployment-dependent, so practitioners should validate the approach under their own workloads before treating them as universal expectations.

Source: Hugging Face Daily Papers