SWE Refactor Bench Tests Whether Coding Agents Can Migrate an Entire Repository

Introduction

Coding agents have become increasingly capable at editing files and repairing localized bugs. Whole-system evolution is a different challenge. A migration that changes a language, build system, or execution target can affect source files, dependencies, configuration, interfaces, tests, and operational assumptions across an entire repository. SWE Refactor Bench is designed to measure that long-horizon capability.

Passing tests is not enough

Many software-engineering benchmarks primarily ask whether the resulting program behaves correctly on a predefined test suite. That leaves room for a simple but important failure mode: an agent can preserve the original implementation, or avoid the requested technology change, while still making the tests pass. The authors call this failure mode “Blindness.”

The benchmark addresses it with a three-stage evaluation process:

• Migration Audit checks whether the requested migration actually occurred.
• Behavioral Tests use a fixed test suite to assess whether the migrated project still works.
• Agentic Verification uses six independent coding agents to generate targeted tests for behavioral differences that the fixed suite may miss.

The benchmark contains 20 whole-repository migration tasks covering four kinds of technical debt. The examples include migrations involving SQLite, zlib, libsodium, and GraphHopper, as well as transitions such as C to Rust, Maven to Gradle, and POSIX to WebAssembly.

Two abilities that do not automatically coincide

The reported evaluation covers 520 runs involving eight frontier models and 26 model-effort configurations. Only 28 runs, or 5.4%, passed all three stages. Thirteen of the 20 tasks received no accepted solution at all. The top-performing model, claude-opus-5, scored 47.0/100.

The breakdown is more informative than the aggregate score. Of the 340 runs that passed Migration Audit, 58% reached 99% of the fixed checks, but only 26% reached 100%. In other words, agents often made the migration substantially complete while still leaving behind a behavioral defect or an overlooked edge case. A few runs preserved behavior by effectively skipping the migration and were stopped at the audit stage; most genuine migration attempts failed later on behavioral tests.

Performance also varied by migration category. Agents scored 31.4 on build-toolchain rewrites, compared with 5.6 on language rewrites. Toolchain changes can often be concentrated in build configuration and project metadata. Language migration requires broader coordination across semantics, dependencies, interfaces, and runtime behavior.

Why the benchmark matters

SWE Refactor Bench draws a useful line between “the code runs” and “the engineering objective was completed.” That distinction matters in production, where a migration is usually undertaken to remove technical debt, reduce maintenance cost, or move to a required platform. A result that merely satisfies visible tests may preserve the old debt while creating new compatibility risks.

For agent developers, the challenge is therefore not just generating code. Agents need stronger repository-wide planning, dependency tracking, migration auditing, coverage analysis, and iterative diagnosis after failed tests. The results suggest that system-scale migration remains an open problem. Current agents may be valuable for bounded parts of a migration, but they are not yet reliable independent owners of the entire change.

Source: Hugging Face Daily Papers