BM25 at Scale: What a New RAG Study Says About Retrieval Choices

Introduction

RAG systems now come in many forms: classic lexical search, dense retrieval, graph-based indexing, and agentic search that explores a file system or knowledge space step by step. But comparisons across these paradigms are often hard to interpret. Many studies use different benchmarks, different corpus sizes, or different judging setups, making it unclear whether a method is genuinely better or simply better suited to a particular scale.

The paper “BM25 Wins at Scale” tackles this question directly. Instead of evaluating each approach at one fixed corpus size, the authors build a controlled scaling study with 28 strictly nested corpus tiers spanning roughly a 450-fold range. The questions remain fixed, as do a bedrock set of relevant and adversarial documents. A single reader model and one judging protocol are used throughout, while the study measures accuracy, construction and query token costs, and latency.

Key takeaways

• The winner depends on corpus scale. At the smallest shared tiers, File-System Agent leads. Its sequential exploration can be useful when the search space is limited and the agent can inspect a meaningful portion of the corpus.
• Agentic exploration is expensive. At the bedrock tier, File-System Agent uses 39 times more query tokens. As the corpus expands, its sequential search becomes less effective because the space grows faster than the agent can explore.
• BM25 crosses over around 10 million corpus tokens. At that point, BM25 overtakes File-System Agent and remains ahead at every larger shared tier. At full scale, its margin approaches 20 percentage points.
• BM25 defines the low-cost end of the Pareto frontier. It does not require LLM-based construction, which makes it attractive for systems that need predictable indexing and query costs.
• Dense retrieval remains efficient but not leading in accuracy. The study finds dense methods computationally appealing, yet less accurate than BM25 under the shared evaluation setup.
• Graph-based RAG faces construction barriers. Some graph approaches hit construction walls before deployment scale, and scalable variants still trail BM25 at shared tiers.

Why it matters

The study’s message is not that BM25 is always the universal winner. Rather, it shows that scale changes the ranking of RAG strategies. Agentic search may shine when the corpus is small enough for sequential exploration to matter. But as the corpus grows, global candidate ranking becomes increasingly important.

For practitioners, this has immediate implications. A more complex RAG pipeline is not automatically a better one. If a system must serve large document collections with reasonable cost and latency, BM25 remains a strong baseline and often a practical default. The paper also suggests a more balanced architecture: use lexical retrieval for broad, cheap candidate discovery, then apply agentic reasoning or reader-model reasoning after the candidate set has been narrowed.

This is a useful reminder for RAG evaluation. Single-point benchmark scores can hide scaling behavior. Real deployments need to consider not only answer accuracy, but also index construction cost, query-token usage, latency, and what happens when the corpus becomes much larger. In that setting, an older lexical method may still be the most scalable foundation.

Source: Hugging Face Daily Papers