RAG Poisoning Leaves a Trace in Attention, Even When Answers Look Right

Introduction

Retrieval-augmented generation has become a standard way to give language models access to fresh knowledge and private or domain-specific documents. Yet the retrieval layer also creates an attack surface. An adversary who inserts carefully crafted material into a corpus or retrieval result can steer the generator toward an unwanted conclusion.

The problem is not always visible in the answer. In fact, a poisoned response may look unusually confident. The study titled “When Context Bites: Detecting RAG Poisoning via Document-Level Attention Collapse” argues that detectors should therefore look beyond the output and examine how the model uses its retrieved context.

Key points

• Output uncertainty can be misleading. Existing defenses often use perplexity, consistency checks, or related output-side signals. Carefully written attack documents can fit the query and surrounding context so well that the resulting response has lower perplexity than a benign response. A confident output is not necessarily a trustworthy one.
• Poisoning changes document-level attention. During benign generation, attention may be distributed across several relevant documents. Under attack, the model’s attention increasingly concentrates on poisoned material. This concentration produces a measurable reduction in document-level attention entropy, which the paper calls “Attention Collapse.”
• D-SCAN examines the generation process. The proposed framework, short for Document-level Signal Collapse Analysis, tracks attention signals assigned to retrieved documents and analyzes how those signals evolve while the response is generated. The approach is designed as a lightweight layer that can complement existing RAG defenses.
• A correct answer does not rule out an attack. According to the paper, D-SCAN can identify cases in which poisoning has not yet changed the final answer. The malicious document may still have gained disproportionate influence over the model’s internal use of context, creating a warning sign for later queries.

Why it matters

The work shifts part of RAG security from outcome inspection to process inspection. A response can be correct while the context supporting it is already compromised. Detecting an abnormal concentration of attention could help systems prioritize source review, trigger additional verification, or combine the signal with output auditing and document trust scores.

The finding should not be treated as a complete explanation of model behavior. Attention patterns vary with model architecture, prompt design, document ordering, and implementation details. The robustness of D-SCAN in multi-turn, agentic, or highly heterogeneous retrieval settings still requires broader validation. In production, attention collapse is better used as one signal among several, alongside source authentication, document deduplication, consistency checks, and response review.

The broader lesson is straightforward: RAG defenses should ask not only whether an answer is correct, but also which document the model trusted most and why it became so certain.

Hugging Face Daily Papers