HyQuant Uses Hybrid Precision to Compress LLM Attention

Introduction

Low-bit quantization can reduce the memory and compute cost of large language model inference, but attention states are particularly difficult to compress. The longer the context, the larger the KV cache becomes. At very low bit widths, however, the resulting error is not necessarily spread uniformly across all tokens. A small number of positions can have a disproportionate effect on attention outputs and, consequently, on long-context quality.

HyQuant addresses this problem with a simple premise: not every attention state deserves the same precision. Instead of applying one quantization format to the entire context, it attempts to identify the regions that matter most and protects them with higher precision.

The observation behind the method

The paper and the accompanying project description report recurring “vertical-line” patterns in attention maps from Qwen3, Llama 3, Gemma 4, and Qwen3.5. These patterns correspond to a limited set of key positions that are repeatedly attended to by many query tokens. According to the supplied material, the top 5% of key positions together with a recent 128-token window account for roughly 82% to 86% of the attention mass.

This observation motivates a mixed allocation of precision. Important vertical-line tokens and nearby context are retained in FP16, while the much larger remainder of the context is quantized more aggressively. The approach therefore targets the concentration of attention rather than treating quantization as a uniform transformation.

How HyQuant works

• Lightweight selection: HyQuant uses vertical-line-aware attention-pattern signals to find tokens that should retain higher precision. The reported identification overhead is around 3% to 5% of runtime.
• Prefill support: During prompt processing, the system keeps selected vertical-line tokens and a local sliding window in FP16, while quantizing the rest of the context.
• Decode-time KV compression: During autoregressive generation, the same principle is applied to the KV cache. Dequantization is fused with attention computation to reduce extra memory traffic.
• Dedicated kernels: Hybrid-precision operators are provided for both prefill and decode, with the goal of avoiding the overhead that can otherwise offset the benefits of quantization.

Reported results

On a single H100, the supplied project description reports that the HyQuant decode kernel can be up to 3.58 times faster than FlashAttention-2 at a 32K context. The end-to-end decode improvement is smaller, at 1.04 to 1.17 times, which illustrates that kernel-level gains do not automatically translate into the same system-level speedup.

For Qwen3-8B in thinking mode, the reported LongBench average is 45.04 for HyQuant, compared with 44.59 for FlashAttention-2. The supplied comparison lists scores from 37.7 to 40.5 for KIVI, SageAttention, and KVTuner. The authors also report that HyQuant still runs with a batch size of 16 and a 32K prefix, reaching 231.6 tokens per second, while the listed comparison systems run out of memory in that setting. Similar results are said to hold for Qwen3-32B, Llama 3.1-8B, and GLM-4-9B.

Why it matters—and what remains open

HyQuant is notable because it reframes attention quantization as a selective precision-allocation problem. A small high-precision budget protects states that appear important, while low-bit storage captures most of the memory savings. This design is especially relevant to long-context serving, where the KV cache can become a major memory constraint.

The reported numbers should nevertheless be read in context. They come from particular models, workloads, sequence lengths, and a single-H100 setup, so they do not establish universal speedups across hardware. Deployment will also need to account for token-selection overhead, kernel support, and whether the observed attention patterns remain stable across models and tasks.

Source: Hugging Face Daily Papers