RAG Retrieval Needs Better Sets, Not Just More Relevant Documents

Introduction

Search ranking was historically designed for humans. A user scans a results page, opens several links, and resolves overlap or disagreement manually. RAG systems and AI agents change that assumption. Search results are increasingly consumed by models, and the documents placed into context define the ceiling of the generated answer.

This paper, “Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Ranking,” focuses on that shift. Its central claim is simple but important: a collection of individually relevant documents can still be a poor evidence set. The set may repeat the same fact, omit a key angle, contain unresolved conflicts, or fail to provide complementary support.

Key points

• From document relevance to set quality: The authors argue that common evaluation pipelines score documents independently and then aggregate those scores through metrics such as nDCG. That approach misses interactions among documents, including redundancy, contradiction, and complementarity.
• SetwiseEvalKit: The paper introduces a benchmark for evaluating document sets across short-form and long-form scenarios. It is organized into three levels and nine dimensions, with roughly 28K high-quality evaluation rubrics.
• Rerankers still struggle: The authors evaluate 12 reranking methods. Even the best method reaches no more than 45% coverage, and dimensions that require cross-document coordination are generally weak.
• No universal winner: A method that performs well in one setting does not necessarily remain best in another. This suggests that short-form and long-form generation require different evidence-selection behavior.
• Rubric4Setwise: The paper proposes a training-free approach that converts rubric-based evaluation criteria into signals for selecting document sets. According to the paper, it achieves the best downstream generation performance while using fewer documents and search rounds.

Why it matters

The work addresses a practical failure mode that many RAG systems encounter. Retrieval scores may look acceptable, but the final context can still be noisy: several passages say the same thing, important evidence is missing, or sources disagree in ways the model cannot resolve. A relevance-centric metric may not reveal these issues, because each document can appear useful on its own.

SetwiseEvalKit makes those problems explicit. Instead of asking only whether each document is relevant to the query, it asks whether the selected set works together as evidence. This is a better fit for model-facing retrieval, where the output depends on the combined context rather than on a ranked list meant for human browsing.

Rubric4Setwise also points to a broader design pattern: evaluation criteria can become optimization signals. Rather than treating rubrics as a post-hoc scoring tool, the method uses them to guide which documents should be selected in the first place.

There are still open engineering questions. The source discussion notes that the rubrics are query-specific and generated using the query and answer page. That can make them more tailored, but it also raises questions about cost, stability, and transfer across domains such as legal search, code search, or enterprise knowledge bases.

Overall, the paper reframes RAG retrieval as evidence-set construction. As AI agents rely more heavily on search and multi-source reading, the next step is not simply retrieving more documents or scoring them one by one, but building smaller, more complete, less redundant, and better coordinated document sets.

Source: Hugging Face Daily Papers