Escalating Weight Precision Only When Decoding Becomes Uncertain

Weight precision is usually fixed for an entire generation. A local model may run fully at low precision to reduce memory traffic, or fully at higher precision to preserve output quality. This paper studies a more selective alternative: use a cheap precision for ordinary decoding steps, then spend extra bytes only when the low-precision result appears unreliable.

The setup stores weights at 8 bits but first decodes with their top four bits. After the low-precision pass, the system checks the output distribution. If its entropy is above a threshold, or if the gap between the most likely and second-most likely token is small, the same step is recomputed using 8-bit weights. The experiments use Qwen3-0.6B on GSM8K, with cost defined as the number of weight bytes read per generated token.

Key findings

• Across 36,724 replayed steps, entropy predicts whether the low-precision argmax will flip after high-precision recomputation with an AUC of 0.93.
• At an escalation budget of about 15%, the entropy gate catches 78% of flips and the margin gate catches 94%. A random gate at the same rate would have a 15% expected capture rate.
• In 150 live problems, all-high precision scores 64.0% and all-low precision 55.3%. An entropy gate that escalates roughly 16% of steps scores 62.7% while using about 0.87 times the all-high weight bytes.
• The paired difference is -1.3 percentage points versus all-high precision, with an interval from -8.0 to +5.3, and +7.3 points versus all-low precision, with an interval from 0.0 to +14.7. The live data do not clearly separate the adaptive method from either endpoint.

The most interesting result is not a demonstrated accuracy win, but a signal for allocating precision selectively. Output uncertainty is highly informative about whether two precision settings will disagree at the next token. That is useful for avoiding unnecessary high-precision reads. Yet an argmax flip is only an intermediate event, not a task-level error. A changed token can still lead to the same final answer after subsequent decoding, so the decisive comparison should be final correctness at an equal byte budget.

Several limitations make the current evidence preliminary. The study does not report a uniform 5-bit or 6-bit decoder at the same cost, even though such a baseline could spend a similar average number of bits without introducing a per-token gate. It also lacks a live random control, a phase-specific rule for thinking traces, a second model, and live KL measurements. Verified speculation is reported to reach zero KL at similar bytes in the same cost model, but that observation does not by itself establish that selective weight precision is the better system.

Serving behavior is another open question. In a single-stream experiment, rereading weights at a higher precision is easy to describe. Under continuous batching, however, one request may trigger an escalation while others do not. The implementation may need both precision formats resident, may introduce synchronization costs, or may reduce batch efficiency. None of these concurrent-serving effects were measured in the supplied results.

The paper therefore presents a promising inference policy rather than a finished optimization recipe. A convincing next step would compare adaptive escalation against fixed 5- and 6-bit baselines under identical byte budgets, evaluate final-answer accuracy over larger and more varied tasks, and report throughput, memory peaks, and latency under concurrent requests. Only then will it be clear whether precision should be spent on difficult decoding steps or distributed uniformly across the entire generation.

Source: Hugging Face Daily Papers