Do Coding Agents Understand Repositories—or Just Recognize Them?

Introduction

Repository-level benchmarks such as SWE-bench have become central to the evaluation of coding agents. Unlike function-completion tests, they ask a model to inspect a real project, identify the source of a bug or missing feature, modify several files when necessary, and validate the result with tests. That makes them much closer to practical software engineering. Yet their realism creates a potential weakness: the underlying open-source repositories may have appeared repeatedly in training data, public patches, or earlier evaluation workflows.

A strong score may therefore combine genuine repository reasoning with recognition of familiar project signals. File names, directory conventions, issue wording, implementation idioms, and known repair patterns can all guide an agent toward the relevant code. The agent may still perform useful reasoning, but a static benchmark cannot easily reveal how much of the result depends on prior exposure.

The SchrodingerRepo approach

Researchers from Shanghai Jiao Tong University propose SchrodingerRepo to probe this question. The framework treats the repository representation as a latent variable that is instantiated when evaluation begins. The agent receives a runnable project whose behavior remains equivalent to the original, but whose surface form is deliberately made less familiar.

The transformations are introduced in four stages:

• Problem statement reconstruction: the task description is reworked so that the agent is less dependent on the original issue wording.
• Namespace remapping: names and identifiers are changed, weakening shortcuts based on familiar classes, functions, or modules.
• Intra-file layout reordering: code organization inside files is rearranged while preserving execution behavior.
• Functionality-preserving rewriting: implementation patterns are rewritten without changing the intended functionality.

The authors evaluate popular language models on SWE-bench Verified and SWE-QA. According to the reported findings, removing familiar repository cues consistently reduces performance across models. It also increases interaction costs. The additional effort is mainly associated with repository exploration and fault localization, rather than only with writing the final patch.

Why it matters

The study does not establish that current agents merely memorize answers. It does, however, show that benchmark scores can mix several capabilities: general debugging, recognition of repository conventions, and memory of projects that may have appeared in training material. When the benchmark representation stays fixed, an agent familiar with a project can potentially reach the relevant location through a shorter route.

A robust coding agent should be able to navigate an unfamiliar directory, build an understanding of dependencies, trace behavior across files, test hypotheses, and narrow the search space even when names and layouts provide little help. Dynamic representations offer benchmark designers a way to separate those abilities from simple repository recognition. Multiple behavior-equivalent versions of the same task could make evaluation less dependent on a single canonical presentation.

The framework also raises an important methodological requirement: transformations must be validated carefully so that they preserve task semantics and do not introduce unrelated obstacles. Its broader contribution is to turn a suspected leakage problem into an experimental question. High performance on a familiar codebase is useful evidence, but it should not be treated as conclusive proof that an agent can operate reliably in an unfamiliar software environment.

Source: Hugging Face Daily Papers