How QAH Helps a 4-Bit Model Beat Its BF16 Checkpoint

Introduction

Model compression usually trades capability for efficiency. A deployment pipeline may first remove layers, attention heads, or neurons, and then quantize the remaining weights to 4 bits. This reduces memory and compute requirements, but it can also weaken reasoning, mathematics, and code generation. A Hugging Face Blog post presents Quantization-Aware Healing, or QAH, as a different way to recover those losses.

What changes in QAH

• The teacher is the original model. Conventional quantization-aware distillation typically teaches a quantized student to imitate the recovered BF16 version of the compressed architecture. QAH instead uses the pre-compression, full-size, full-precision model as the teacher. The architectures do not need to match because the transfer is based on output logits.
• Quantization becomes another distillation pass. Rather than treating quantization as a final lossy conversion after recovery, QAH uses it as an opportunity to transfer information that the earlier compression and healing stages may not have captured.
• Long-context training is memory-aware. The method uses a chunked KL-divergence loss. It processes portions of a sequence at a time instead of materializing the full sequence-by-vocabulary tensor, allowing healing on documents of up to 32k tokens within a fixed GPU memory budget.

Reading the results

The team compressed GPT-OSS 120B to 60B, recovered a BF16 checkpoint, and then produced a 60B MXFP4 model with QAH. Against the same architecture in BF16, the QAH model won on seven of nine evaluations. The largest improvements appeared in capabilities that structural compression often harms: AA-LCR long-context reasoning rose by 7.4 points, while AIME 2025 mathematics rose by 5.6 points. The model also improved on agentic coding, tool use, science question answering, instruction following, and LiveCodeBench.

The two declines were small in absolute terms: MMLU-Pro fell by 0.2 points and SciCode by 1.4 points. The 60B QAH model also edged past the 120B teacher on LiveCodeBench, scoring 66.5 versus 66.0. That is not a universal victory over the larger model, however. On GPQA Diamond it scored 67.4, below the teacher’s 69.0, and the remaining gap was especially visible on the demanding long-context evaluation.

Why it matters—and what remains open

QAH changes the role of the recovery stage. A compressed model is no longer constrained to imitate an intermediate checkpoint that has already lost capacity and information. It can continue learning the behavior distribution of the original model while operating with fewer parameters and 4-bit weights. This could be useful for memory-constrained inference and cost-sensitive serving.

The evidence also needs careful framing. The reported results come from one GPT-OSS setup, one MXFP4 quantization configuration, and a limited set of benchmarks. The material does not establish how training cost, data requirements, or results would change across architectures. QAH is therefore best understood as a promising recovery recipe—not proof that 4-bit models generally outperform their BF16 sources.

Source: Hugging Face Blog