CrossSWE Shows Why Coding Agents Struggle Across Repositories

Introduction

Coding agents have made rapid progress on software-engineering benchmarks built around a single repository. To make those tasks harder, researchers have often increased the number of edits, extended the execution horizon, or expanded the available context. Real development, however, introduces another kind of difficulty: a single feature or bug fix may require coordinated changes in several dependent repositories.

Updating a client without changing a service interface, or modifying a library while overlooking its callers, can leave every local patch looking plausible while the overall system remains broken. CrossSWE is designed to measure precisely this gap. It treats joint development across repositories as the task, rather than treating a successful patch in one codebase as sufficient.

Key points

• Broad coverage: CrossSWE contains 171 real-world instances across 165 repositories, 20 technical domains, and 13 programming languages.
• Coordination is required: The benchmark checks whether modifying only a subset of the involved repositories is insufficient to produce the intended behavior.
• Performance remains limited: Five LLMs were evaluated, and the strongest reported result came from GPT-6.1 Sol at 58.48% Pass@1.
• Consistency is the central failure: Cross-repository inconsistencies represented 44.2% of observed failure modes, exposing difficulties with interfaces, dependencies, and propagated changes.

Why it matters

A single-repository task lets an agent search one code tree, edit files, and run a relatively local test suite. A cross-repository task requires a system-level model before implementation begins. The agent must discover which projects share contracts, data structures, or version assumptions; determine how a change should propagate; and decide what must be tested together.

The success criterion therefore changes from local patch correctness to system consistency. This distinction is important for both benchmark design and tool development. Future evaluations may need to measure dependency discovery, interface evolution, version coordination, and integration testing—not just whether generated code passes tests in its immediate repository.

CrossSWE’s contribution is not simply another score for coding models. It turns a common condition of real software development into an explicit, measurable challenge. The results suggest that improving coding agents will require more than larger context windows or longer action sequences. They will also need better repository mapping, impact analysis, and mechanisms for checking whether changes remain aligned across project boundaries.

Source: Hugging Face Daily Papers