Back to articles

QDA Uses Agents to Search for Better LLM Quantization Recipes

3 min read

Quantization can make large language models cheaper to serve, but choosing how to quantize them is not a one-size-fits-all exercise. The best combination of methods can change with the model and target precision, leaving experts to repeat substantial manual tuning. QDA, short for Quantization Design with Agents, proposes an alternative: an agent iteratively builds and evaluates post-training quantization recipes, then uses the results to choose what to try next.

A faster feedback loop

The system follows a familiar engineering cycle—propose a recipe, build a candidate, measure it, and revise the design. Its central contribution is to make this loop cheaper, so a search can examine more alternatives within a practical time budget.

  • Screen candidates with KL divergence. QDA compares output distributions from a quantized model and its original counterpart on reasoning traces that the agent does not see. A score takes minutes, rather than the tens of GPU-hours that evaluating a quantized reasoning model can require.
  • Check promising candidates with long generations. A fast distributional score is useful for ranking, but it does not by itself establish task performance or catch every generation failure. QDA therefore runs longer generations on selected candidates to look for problems such as repetition loops.
  • Make methods composable. The system includes a library of 21 public post-training quantization methods that can be combined in different orders and adapted to new model architectures. The paper says a search can assess up to 36 candidates in an eight-hour window.

Reported results and practical implications

Across four reasoning models, QDA improves average accuracy over the strongest public recipe by 0.9 percentage points at W4A16, 1.4 at W4A8, and 2.2 at W3A16. Each search uses 64 GPU-hours. For the 320-billion-parameter GLM-5.3-Flash at NVFP4 (W4A4), a 6.5-hour search costing $69 reaches 98.3% of the FP8 model's accuracy, compared with 96.9% for the best vendor checkpoint cited by the paper. The authors also report that the same agent, when given NVIDIA ModelOpt tools instead of QDA's tools, trails by 0.5–1.3 points on Qwen3.6-35B. That comparison underscores that the search infrastructure matters alongside the agent.

The broader idea is to turn model-specific quantization from repeated expert hand-tuning into a reusable, automated search process. It does not remove the need for careful validation: KL divergence is a screening signal, and long-generation checks remain important. Nor do results on these models establish that the same gains will hold across every architecture or workload. The paper says QDA checkpoints run on existing serving kernels without inference overhead, while agent-token costs range from $19 to $75 per search. If the reported compatibility generalizes, this approach could lower the engineering effort needed to deploy low-bit models, while still requiring GPU time for the search itself.

Source: Hugging Face Daily Papers

Comments

Checking sign-in status...

Loading comments...

Related articles