Why RAG Answers Change After Index Expansion

Retrieval-augmented generation is often treated as a model connected to an external knowledge base. That framing encourages a simple expectation: when the index grows, the system should gain evidence while preserving answers to questions it already handled. A study highlighted by Hugging Face Daily Papers challenges that assumption. It finds that expanding the retrieval corpus can change an agent’s answer even when the requested model identifier, prompt, retrieval policy, evidence depth, rendering procedure, and exposed generation controls are held constant.

The authors describe this as “accuracy-blind answer churn.” Accuracy can conceal the problem because it only records whether an answer matches a target. If an update fixes some questions and breaks others, the gains and losses may cancel out in the aggregate score. Meanwhile, a language model can produce different responses on repeated runs against the same snapshot. A one-shot before-and-after comparison therefore risks attributing normal generation noise to the index update.

To address both issues, the paper introduces the Snapshot Compatibility Audit. Its logic is straightforward:

• repeat generation on the same snapshot to estimate ordinary disagreement;
• compare outputs across the old and expanded snapshots to measure total disagreement;
• subtract same-snapshot disagreement from cross-snapshot disagreement to estimate excess churn caused by the corpus or index change;
• evaluate both normalized exact answers and blinded semantic equivalence instead of relying on string identity alone.

In a preregistered study of 400 Natural Questions items, the researchers expanded a frozen FineWeb prefix from one shard to seven. Excess churn reached 6.44 percentage points under normalized exact matching and 10.25 points under blinded semantic evaluation. Yet exact-match accuracy changed by only -1.50 points. A post-hoc analysis identified repeat-stable semantic flips on 40 of the 400 questions: the answers were stable when the same snapshot was queried repeatedly, but changed meaningfully after the index expansion.

A separately preregistered 200-question TriviaQA study produced smaller, directionally consistent excess churn, while exact-match accuracy moved in the opposite direction. An outcome-blind post-hoc replication on 100 questions, using a second DeepSeek generator and a different serving configuration, found 8.75 percentage points of semantic excess churn even though exact-match accuracy increased by 3.00 points. These results do not show that index expansion always reduces quality. They show that utility and compatibility are different properties.

The engineering implication is important. A RAG release can improve its benchmark score while silently changing many existing answers. Products that require predictable behavior, evaluation suites intended to track regressions, and agents used in workflows should therefore record corpus snapshots, inspect retrieved evidence, compare answer changes, and measure repeat stability. Future release reports should place answer compatibility beside accuracy, not treat accuracy as its substitute. New knowledge is valuable, but teams also need to know which established answers changed and whether those changes were intended.

Source: Hugging Face Daily Papers