Better Retrieval, Worse Robustness: How Multi-hop RAG Amplifies ASR Errors

Introduction

Speech-based question answering usually places automatic speech recognition before retrieval. By the time a RAG system receives a query, the text may already contain substitutions caused by pronunciation, accent, or recognition uncertainty. This creates a difficult systems question: can multi-hop retrieval structures absorb those mistakes, or do they make them more damaging?

The paper Better Retrieval, Worse Robustness presents evidence for the latter. Entity-aware linking and iterative reasoning can improve the absolute quality of a system, yet the same mechanisms may amplify errors that enter at the speech-recognition stage.

Key findings

• The evaluation studies two common extensions to standard RAG: entity-graph linking and iterative query reformulation.
• Experiments use HotpotQA, 2WikiMultiHopQA, and MuSiQue, together with four English accents synthesized through neural TTS and passed through ASR.
• Four RAG configurations are compared with a clean-text oracle. Structurally richer systems generally maintain higher absolute F1 with ASR input, but lose more of their clean-text advantage.
• Across all three benchmarks, the clean-text-to-highest-WER F1 gap is 36%–67% larger for the combined IRCoT+HippoRAG2 setup than for naive dense retrieval.
• On 2WikiMultiHopQA, corruption of one or more query entities accounts for 87%–96% of degradation cases across the four methods.
• N-best decoding and phonetic entity correction provide only limited relief, leaving most of the performance gap in place.

Why structure can amplify noise

Multi-hop retrieval depends on identifying a useful entity and then following evidence or relations to the next step. If ASR changes a person, place, organization, or work into a plausible but incorrect surface form, an entity linker may select the wrong node. An iterative reasoner can then reformulate the next query around that mistaken premise, turning one transcription error into a chain-level retrieval failure.

Naive dense retrieval is less capable in some clean-text settings, but its shorter processing path may also provide fewer opportunities for an early entity error to become structurally reinforced. The result is an important distinction between absolute performance and robustness: a method can score higher on noisy input while still suffer a larger drop relative to its ideal text condition.

Implications

The findings do not show that graph augmentation or iterative reasoning is useless for speech applications. They show that text-oriented retrieval improvements cannot be assumed to transfer safely to voice interfaces. A practical system may need to expose ASR uncertainty to retrieval, maintain candidate entities rather than committing immediately, and verify important names before graph traversal or multi-step reformulation.

The study also argues for a more diagnostic evaluation protocol. Reporting only final F1 under one ASR configuration can hide whether a system is genuinely robust or simply stronger overall. Clean-text comparisons, accent variation, and entity-level error analysis reveal where the performance is lost. The released resource covers three benchmarks and four accents, with 3,000 questions and 12,000 transcript rows, enabling further work on this problem.

For speech RAG, the central design goal is therefore not merely deeper retrieval. It is controlled propagation of uncertainty: the system should know when the entity driving its multi-hop search may not be the entity the user actually spoke.

Source: Hugging Face Daily Papers