RidgeFill Speeds Up Long-Context Prefill Without Retraining

Introduction

For long-context language models, the time before the first generated token can dominate the user experience. The main cost is often Prefill: processing the entire prompt through every transformer layer before decoding starts. RidgeFill proposes a training-free approximation that reduces this cost without simply dropping prompt tokens or modifying the base model.

How it works

The method divides the network at a chosen split layer. Lower layers process the prompt exactly as usual. Starting from the hidden state at that split, a separate map predicts the input that each later layer would have received. The predicted inputs are then used to produce the late layers’ cache entries.

The maps are fitted with ridge regression in closed form from one forward pass over calibration text. There is no gradient-based training and no update to the model weights. The late layers’ own projections remain unchanged. RidgeFill also keeps full computation for selected rows, the last few thousand prompt tokens, and every decoding step. These safeguards make the approach a selective approximation rather than a blanket layer-skipping scheme.

The design follows ideas associated with YOCO and DeepSeek-V4.1-Flash, where similar projection behavior is incorporated during model training. RidgeFill instead attempts to retrofit the behavior onto models that were not trained for it.

What the measurements show

A central contribution is the evaluation methodology. Loss averaged over all prompt tokens can hide errors in the filled region, so the authors use several targeted tests: needle retrieval inside that region, loss on tokens dependent on distant context, and splice tests that replace predicted keys or values with exact ones. These experiments suggest that the dominant error source is model-dependent. Keys matter most for Llama-3.1-8B, with their scale accounting for a substantial portion of the effect, while values and recurrent state are more important for Qwen3.5-9B.

In a llama.cpp fork running on an A100, Qwen3.8-27B reaches 1.25–1.31× faster time to first token for prompts from 16K to 120K tokens. The reported saving is about 28 seconds for a 120K-token prompt when the maps fit in VRAM. RULER changes by roughly -1.2 to -1.4 points, and LongBench v2 by about -1.4 points. The 95% intervals include zero, but they do not rule out losses of a few points. In vLLM, Qwen3.8-Flash-Next reaches 1.50–1.60× speedup over exact Prefill for 9K–32K-token inputs.

Why it matters

RidgeFill offers a practical route to experimenting with approximate Prefill on already released models. Its main appeal is the combination of a one-time calibration procedure, unchanged model weights, and available llama.cpp support. However, the maps’ behavior on prompts unlike the calibration data remains an important deployment question. Performance also depends on context length, framework, hardware, and how many tokens are kept on the exact path. The results therefore position RidgeFill as a tunable speed–quality trade-off, not a lossless replacement for full Prefill.

Source: Hugging Face Daily Papers