Code-search benchmarks often put dense encoders well ahead of BM25. Yet developers and coding agents routinely reach for grep. A new study asks whether this gap reflects a fundamental weakness in lexical search—or partly reflects the way code-search benchmarks are built.
The researchers introduce Repo-Premium, a benchmark designed around file-level retrieval in real repositories. It covers Python, JavaScript, TypeScript, Go, Rust, and Java, with pull requests from 2022 through 2026 and a preregistered test pool spanning more than 91 repositories. The goal is to move beyond curated queries and older code, and to evaluate systems in a setting closer to repository work.
The paper identifies several evaluation choices that can tilt results. Splitting files into chunks for BM25 changes the conditions under which lexical matching operates. Treating edited test files as relevant can also affect what counts as a successful result. Query wording matters too: when a query omits a rare identifier used in the code, a term-matching system has little to latch onto, while a dense model may still retrieve something based on semantic similarity. That can make the latter look more capable than it would against a fairer lexical baseline.
To test the role of missing terms, the authors try “generate-then-grep”: an LLM proposes identifiers likely to appear in the code, then lexical search uses the expanded query. This helps BM25 much more than it helps dense retrieval. With the expanded queries, BM25 matches dense retrieval in every Python pool, including on pull requests opened after the LLM was released. This result is specific to the Python pools studied; it should not be read as proof that the same outcome holds across all six languages. The researchers also report that none of the methods they tested could reliably predict which retriever would win for an individual query.
Key findings
- Dense-retrieval gains should be measured against a strong lexical baseline, with chunking and relevance judgments made explicit.
- When a query lacks rare names from the code, adding those terms may matter more than switching retrieval models.
- The approaches can complement each other. On 539 lockbox queries, fusion of the top-ten results from both retrievers reached 0.551 nDCG@10, compared with 0.533 for dense retrieval and 0.432 for BM25.
The study does not establish that BM25 is generally better than vector search. Instead, it challenges the assumption that benchmark rankings automatically predict what works in real development. Code contains both semantic clues and highly specific tokens—function names, variables, and APIs. Search systems should account for both, and evaluations should reflect the repositories and queries developers actually face. For coding agents, the practical takeaway is less about choosing “embeddings or grep” than combining retrieval signals and supplying the words that connect a request to the code.
Source: Hugging Face Daily Papers
Comments
Checking sign-in status...
Loading comments...