Why Gated DeltaNet Holds Up Under 4-Bit Quantization

Introduction

Hybrid language models combine softmax attention with linear-attention mechanisms such as Gated DeltaNet (GDN). Attention offers flexible token-to-token interaction, while GDN maintains a fixed-size recurrent state that summarizes prior context. That state has made the recurrent half appear especially risky to quantize: an error introduced at one step might, in theory, be carried through thousands of later updates.

The work featured by Hugging Face Daily Papers tests that assumption on Qwen3.8-27B, a hybrid model with 48 GDN layers and 16 attention layers. Instead of protecting the recurrent block, the authors quantized all 496 linear layers with NVFP4 W4A4. The process used calibration-only post-training quantization, with no quantization-aware training or distillation.

Main findings

• Quality remained close to BF16. The fully quantized recipe was evaluated with perplexity at 4K and 32K, along with MMLU-Pro, GSM8K, AIME'25, GPQA-Diamond, and LiveCodeBench. Its results were reported to remain within seed-level variation of BF16, with a five-task average difference of -0.52. RULER retrieval testing extended to 64K context.
• The gates were less fragile than expected. GDN decay and write-strength projections use softplus, exponential, and sigmoid parameterizations. These nonlinearities compressed an approximately 11% GEMM error to about 2% at the gate output, making the supposedly sensitive projections among the least affected components.
• Recurrent noise did not compound. The delta rule writes along the current key direction and overwrites the state rather than simply adding an unchanged error forever. In the reported study, an injected state impulse was forgotten within hundreds of steps, while noise reached a flat plateau over 32K tokens. The perplexity gap also narrowed with position.
• Block scaling mattered. NVFP4 uses 16-element block scales, which helped localize extreme residual-stream outliers and equalize activation error across different layer roles.
• Serving kernels can introduce a hidden failure mode. Fusing GDN projections into one GEMM while retaining independently calibrated module scales can create a global-scale mismatch. The authors document a correction, and the released checkpoint is pre-harmonized.

Why it matters

Under a common vLLM, TP=1 setup on one 96 GB Blackwell GPU, the Minima checkpoint is about 17.5 GiB, roughly 2.9 times smaller than the BF16 model. Its prefill throughput was reported to be 14% to 19% higher than the compared recipes that retained GDN and attention at higher precision.

The broader lesson is not that every recurrent architecture is automatically safe at four bits. Rather, quantization behavior depends on the interaction between numerical format, nonlinear gates, and the state-update rule. A recurrence that overwrites information along changing key directions may respond very differently from a system that continuously accumulates an error.

The results are specific to the tested model, hardware, kernels, and calibration procedure. Deployers should still verify long-context quality, fused-kernel scale handling, and framework compatibility. Even with those caveats, the study provides a strong case for evaluating hybrid models end to end instead of assuming that their recurrent layers must always be protected with higher precision.

Source: Hugging Face Daily Papers