X-AuT Compresses Speech Audio Encoders Through Progressive Distillation

The computational cost of a speech language model is determined not only by its language-model backbone. The audio encoder, or audio tower, also contributes to inference work, especially when it contains many transformer blocks. X-AuT addresses this cost by treating encoder compression as a restoration and distillation problem rather than as a simple delete-and-run operation.

What the framework does

Removing complete encoder blocks can change the embeddings delivered to the decoder. According to the paper, these changes may produce deletion errors or cause the model to emit an end-of-sequence token too early. X-AuT therefore begins with short behavioral probes that help identify useful combinations of layers before a pruning configuration is selected.

The compressed model is then trained with several complementary signals:

• Representation alignment keeps the student’s representations close to those of the teacher;
• Cross-scale distillation transfers information across different representation levels instead of comparing only final predictions;
• Scheduled student-policy supervision gradually introduces the student’s own decoding behavior into training;
• LoRA adaptation updates attention adapters and the tied output embedding while keeping the language-model backbone frozen.

The data pipeline is part of the design as well. Training uses the highest-agreement tier from a transcript-consistency process, and the sources are reweighted during fine-tuning. This suggests that X-AuT is designed around both structural compression and control over the quality of the supervision used to recover the student model.

Reported results

On ten public Chinese-English benchmarks, reducing the audio encoder in Qwen3-ASR-0.6B from 18 to 16 layers lowered the macro-average error from 5.61% to 5.27%. A 14-layer version reduced audio-tower parameters by 20.7% and reached a 5.75% mean error. These results describe two practical operating points: the 16-layer configuration delivered the lower reported error in this experiment, while the 14-layer configuration provided greater parameter savings.

The comparisons also show why the training path matters. With the matched recipe, a 1.7B teacher produced a 5.55% mean error, whereas self-distillation reached 8.45%. Progressive pruning from 18 to 14 layers achieved 5.75%, outperforming direct pruning to 14 layers at 6.73%. The reported figures come from single runs, and benchmark-level effects were not uniform, so they should be interpreted as evidence for a promising compression recipe rather than a universal guarantee.

Why it matters

X-AuT reframes audio-encoder compression as a coordinated process involving layer selection, behavioral testing, representation recovery, and parameter-efficient adaptation. For deployment-oriented speech recognition systems, this may provide a more controllable trade-off among accuracy, model size, and inference cost. The remaining questions are practical: how much end-to-end latency is saved on specific hardware, how stable the results are across repeated runs, and whether the same recipe transfers to other speech models and languages.

Source: Hugging Face Daily Papers