Where ASR Hallucinations Begin: The Encoder’s Final Stage Is a Critical Boundary

Introduction

Errors in automatic speech recognition are not always simple mishearings. Under some conditions, a recognizer can produce text that is fluent and coherent while having little or no connection to the speech that was actually recorded. These outputs are commonly described as ASR hallucinations. Their danger lies precisely in their fluency: a polished transcript can appear more trustworthy than an obviously corrupted one.

A new arXiv study approaches the problem from a more basic angle. Instead of asking how one particular hallucinated sentence is generated, the authors investigate where an encoder can no longer preserve sufficient audio guidance, and where the network turns an abstract acoustic representation into information that its decoder can reliably interpret.

Key findings

• Two decoder families were tested. The study examined two independently trained Conformer-Large recognizers: one using CTC and the other using RNN-T. The models were evaluated under environmental degradation and shifts in speaker and background conditions.
• The final encoder stage acts as a critical boundary. Bypassing the last encoder block caused the output to diverge on nearly every utterance. Skipping middle blocks had much less effect. This suggests that the functional role of the encoder is not uniform across depth, and that the final stage performs an important conversion from abstract acoustic features to text-relevant structure.
• Representations become more compact and text-readable. At this terminal stage, the representations are more compact, the trained decoder can read text from them, and grapheme-related information becomes explicit.
• Breaking the stage does not directly produce fluent fabrication. The intervention mainly yielded garbled or repetitive text. It did not by itself create the smooth, plausible but audio-unrelated output associated with natural hallucinations.

Why it matters

The paper’s most important qualification is that final-stage failure should not be treated as the complete origin of hallucination. More precisely, it is a mechanistic precondition: when the encoder fails to produce an output sufficiently grounded in the audio, the recognizer becomes vulnerable to losing contact with the input. Turning that weakened grounding into fluent fabricated language may require additional factors, such as decoder preferences, language-model priors, input quality, or other properties of inference.

This framing offers a practical direction for model diagnosis. Rather than treating hallucination as an undifferentiated failure under noise or distribution shift, researchers can inspect whether speech evidence remains measurable through the encoder and pay particular attention to the terminal stage. The fact that similar dependencies appeared in both CTC and RNN-T systems also suggests that the observation is not limited to one decoding strategy.

At the same time, the study does not show that repairing the final block would eliminate every hallucination in deployment. Nor does it reconstruct the full chain by which a loss of grounding becomes fluent invented text. The safer interpretation is that the final encoder stage functions as a gate for grounded recognition: when it fails, the audio’s constraint on the transcript weakens sharply. Explaining what happens next remains an open research problem.

Source: arXiv