When Quantized Models Fail on Arrival: A Functional Audit of GGUF Files

Introduction

Quantization has become one of the main ways to run large language models on local hardware. Smaller weight files reduce memory requirements and make consumer deployments practical. Yet a file that downloads, loads, and produces output is not necessarily a functional model. A study featured by Hugging Face Daily Papers examines that gap through an audit of publicly available GGUF artifacts used with llama.cpp-compatible runtimes.

What the audit found

• The researchers tested 305 eligible code-capable tags across 15 model lines in the official Ollama library, plus 22 popular community files hosted on Hugging Face.
• A 15-task coding screen was used to identify suspicious artifacts. Those candidates were then evaluated on all 164 HumanEval+ tasks.
• Seven official tags solved at most six tasks, representing about 2.3% of the official sample. Every one was a low-bit quantization of a model with no more than four billion parameters.
• Five suspicious files were rebuilt from the original weights. Each rebuilt version matched its registry counterpart tensor for tensor, ruling out ordinary file corruption as the explanation.

The most important result came from controlled reconstruction. With the weights, toolchain, and tensor mix held constant, adding an importance matrix improved three Qwen2.5-Coder-3B builds from zero solved tasks to between 76 and 113, depending on the quantization level. This points to calibration as a critical factor in some low-bit pipelines. However, the same uncalibrated recipe did not break the smaller Qwen2.5-Coder-1.5B model. The evidence therefore does not support a simple rule that one recipe or one bit width always fails. Model scale and architecture appear to interact with the quantization process.

The audit also reported that none of 66 sampled tags that passed the initial screen turned out to be non-functional. That result suggests a lightweight screen can be useful as a practical gate, even though it is not a guarantee of universal quality.

Why it matters

The study separates reproducibility from usability. A model can be downloaded intact, loaded successfully, and regenerated exactly while still failing real tasks. Hashes, file-format checks, and build scripts verify the delivery and construction process; they do not verify that the resulting model retains useful capabilities.

For users, low-bit models and smaller parameter counts deserve extra caution. A successful launch should be followed by a compact task-based check, especially when the model is intended for code generation. For registries and publishers, the authors recommend screening tags before release, then calibrating or withholding artifacts that fail. They provide audit data, the quantcheck tool, and rebuild scripts to support this workflow.

The findings do not show that quantization as a whole is unreliable, nor do they establish that every format has the same failure mode. Their narrower lesson is operational: public model catalogs need functional validation in addition to artifact integrity checks. A model is not ready merely because it can be downloaded and loaded.

Source: Hugging Face Daily Papers