SteerDistill Teaches Quantized Agents to Correct Their Own Failures

Introduction

Low-bit quantization is an effective way to reduce the inference cost of large language model agents. Yet agentic workloads are unusually sensitive to small behavioral errors. If an agent misses a task constraint, misreads the current environment, or takes the wrong tool action, the resulting state transition can push the entire trajectory away from success. SteerDistill targets this failure mode rather than treating quantization loss as a generic drop in language quality.

A closed loop around the student’s failures

The paper argues that standard offline distillation has a practical weakness: its data may not contain the contexts in which the quantized student actually fails. Asking the student to produce its own corrections is also unreliable, because the same limited model may have few successful responses available in those situations.

SteerDistill therefore begins with the student’s own rollouts. Its process can be summarized as follows:

• Find a behavioral divergence. The quantized agent is executed in an environment, and the method identifies a response or transition that departs from a successful full-precision reference.
• Steer a retry. A frozen BF16 teacher examines the failure context and guides the student toward a revised response.
• Check the result in the environment. The revised action is executed rather than accepted solely because its text appears plausible. If the behavior is still incorrect, the teacher updates its guidance using the new execution feedback.
• Distill away the guidance. During context distillation, the student matches the steered student at corrected turns and the full-precision teacher at other turns. At inference time, the resulting model runs without the teacher or steering prompts.

This is the central distinction of the approach. The teacher is not simply generating a complete ideal trajectory for every task. It is intervening where the quantized student has demonstrated a concrete problem, then using execution to verify whether the local fix works. The supervision consequently reflects task state and action consequences, not just token-level similarity.

Results and implications

Across five agentic benchmarks and three models, SteerDistill improved average scores over the strongest quantized baseline by 1.67 to 2.18 percentage points under W4A16. Its results remained within one point of BF16. On τ²-Bench, it exceeded QeRL by 1.84 percentage points in success rate while using 67% fewer training GPU hours, and successful trajectories were shorter than those of the quantized baselines. In a W2A16 experiment with Qwen3-8B, it achieved the highest average score among the tested quantized methods.

The findings suggest that some quantization failures are not simply lost knowledge. They are behavioral deviations at critical turns: overlooking a requirement, choosing an unsuitable action, or creating the wrong environment state. If those deviations are found in the student’s own rollout and corrected with feedback, the improvement can be internalized through distillation.

The method still relies on a full-precision teacher and an executable environment during training, so its cost and applicability depend on how easily task behavior can be verified. Even so, it offers a useful direction for deploying compact agents: instead of only imitating final answers, train the model to repair the decisions that actually cause failure, then let it run independently.

Source: Hugging Face Daily Papers