ExecRetrieval Shows Why Similar Code Is Not Necessarily Correct Code

Introduction

Code retrieval is often treated as a relevance problem: given a query, can a system return code about the same topic or task? For coding agents and retrieval-augmented code generation, that definition is incomplete. A retrieved snippet may look highly relevant while failing on the very behavior the user needs. ExecRetrieval, presented in the paper ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval, is designed to measure that distinction directly.

The benchmark asks a focused question: when a search pool contains both a correct implementation and a nearly identical implementation with one behavior-breaking edit, can an embedding-based retriever reliably put the correct one first?

Key points

• Controlled counterfactuals are part of the search pool. ExecRetrieval covers 939 Python tasks. Each task has one canonical implementation verified by execution and up to four buggy distractors. The distractors are produced through mechanical, single-target mutations and then execution-verified as incorrect.
• The benchmark tests functional discrimination. Because the candidates are near-clones, success cannot be explained simply by topic overlap, lexical matching, or identifying a familiar code sample. The ranking itself must reflect functional correctness.
• Top-k recall masks a ranking problem. The study evaluates 23 dense embedding configurations together with BM25. The best hosted system reaches exec@10 of 1.00, yet its exec@1 is only 0.331. In other words, the correct implementation is found within ten results, but frequently is not the first result.
• The wrong first result is usually a paired bug. Across the four leading systems, 91.5% to 99.4% of rank-one failures are the query’s own near-clone buggy variants. In 67% to 78% of queries, the canonical implementation scores below at least one of its four paired distractors.

Why it matters

These results describe a functional-correctness gap in code embeddings. A representation may capture the purpose, structure, and surface meaning of a program without reliably recognizing that a tiny edit changes its behavior. For coding agents, this is more consequential than an ordinary relevance error: the top-ranked result is likely to be inserted into context, used as a template, or passed to a later generation step.

The findings also challenge a common evaluation shortcut. A system can appear strong when the metric only asks whether a correct item appears somewhere in the top-k list. Increasing k may recover the canonical implementation, but it does not solve the decision problem faced by a downstream agent that privileges the first result. Exec@1 and exec@10 therefore measure different operational properties and should be reported together.

ExecRetrieval offers a practical foundation for improving that situation. Retrieval pipelines could add execution checks, test-aware reranking, or a dedicated functional discriminator after vector search. The benchmark also provides a controlled way to determine whether such changes genuinely separate correct code from its closest incorrect alternatives.

The paper has been accepted to the EMNLP 2026 main conference. Its central contribution is not a claim that one embedding configuration has solved code retrieval, but a sharper measurement target: for executable code, semantic similarity is not the same as correctness, and evaluation should make that difference visible.

Source: Hugging Face Daily Papers