Specter Uses Quantized Drafting to Speed Up MoE Expert Offloading

Mixture-of-experts models activate only a subset of their experts for each token, but the full collection of expert parameters can still exceed the memory available on one GPU. A straightforward workaround is to keep some experts in host memory and move them to the accelerator on demand. The problem is that routing is data-dependent: the runtime may not know which experts are required until the current computation has produced its routing decision. PCIe transfers then sit directly on the critical path, adding latency to every generated token.

Specter addresses this problem by extending speculative decoding beyond its usual role of reducing verification work. Its runtime uses a lightweight, quantized draft model on the device to look several steps ahead and estimate which experts the target model will activate. It can then begin moving those experts from host memory before verification reaches the corresponding steps. If the predictions are sufficiently faithful, communication and execution can overlap instead of running as a sequence of wait-and-compute stages.

Key points:

• Quantization enables the predictor: The draft model is made small and efficient enough to run on the accelerator while leaving room for the main model and active experts.
• The system prefetches across multiple steps: Rather than waiting for each current routing decision, Specter uses a lookahead window to organize future expert movement.
• Activation fidelity matters: The draft model must preserve the target model’s expert-selection patterns. Incorrect predictions can reduce the benefit, but the supplied abstract does not specify the miss rate or the cost of a miss.
• The target scenario is constrained hardware: The work focuses on single-request, single-GPU inference, where a model may otherwise be impractical because its experts do not fit in accelerator memory.

According to the abstract, Specter improves time per output token by an average of 1.50x and by as much as 1.88x compared with the state-of-the-art system under single-request inference. The result highlights an important systems insight: MoE offloading is not only a capacity problem. It is also a scheduling problem. If the runtime can anticipate future expert demand, it has a better chance of hiding PCIe traffic behind useful computation.

Several questions remain open from the available material. The paper summary does not provide the expert-prediction accuracy, the slowdown caused by wrong predictions, or measurements across different PCIe links and host-memory technologies. It also does not establish how the method behaves under concurrent requests, where memory contention and changing routing patterns could affect prefetching. Even so, Specter presents a clear design direction: use an inexpensive quantized predictor to turn reactive expert movement into proactive staging, making memory-constrained MoE inference more practical on a single accelerator.

Source: Hugging Face Daily Papers