SWE-Duel Turns Coding-Agent Evaluation into a Self-Scaling Arena

Why static benchmarks stop scaling

Most software-engineering benchmarks are fixed when they are released: the repositories, issues, patches, and tests are known in advance. Over time, parts of that material can enter training data. Once a model becomes sufficiently capable, it may also saturate the benchmark rather than reveal meaningful differences between systems. SWE-Duel, presented by researchers from Singapore Management University, explores a different design: generate the challenge and its verification procedure while the evaluation is running.

A Red-versus-Blue protocol

Two coding agents take turns as Red and Blue inside real repositories. In an isolated session, Red studies the project, proposes a new feature, implements it, and adds feature tests. Red then places a subtle bug in the feature and creates sealed tests that trigger it. The pull request is accepted only when it passes the feature tests and the repository’s existing tests, while failing exactly the hidden bug tests.

The protocol adds a safeguard against impossible challenges. A fresh session of the same agent must be able to repair the defect without being told where it is. If that blind repair fails, the task is discarded. This prevents the arena from rewarding defects that are merely broken or unsolvable rather than genuinely difficult.

Blue receives the resulting pull request without being told that a bug exists. It must inspect the change, reason about the surrounding code, and submit a revision. A win requires passing all three test suites: the feature tests, the original repository tests, and the hidden bug tests. The agents then switch roles. This makes the benchmark evaluate more than code generation: it also measures test interpretation, review, debugging, and the ability to detect problems that are not explicitly announced.

What the tournament found

The reported tournament involved 11 model-and-harness participants, 15 repositories, five programming languages, and 1,500 turns. The leading systems excelled in different parts of the game. GPT-5.5 with Codex CLI ranked first as a solver, repairing 98.3% of the defects it reviewed. Claude Opus 4.8 with mini-swe-agent ranked second overall and led the task-authoring dimension, achieving a 27.3% Red win rate.

A comparison within one model family offers preliminary evidence that the arena can scale with agent capability. Tasks authored by the newest version defeated a fixed reference solver 42.9% of the time, compared with 4.0% for tasks authored by older versions. The result suggests that stronger agents may produce more discriminating challenges instead of simply repeating familiar benchmark patterns.

Why it matters—and what remains open

SWE-Duel reframes a benchmark as an ongoing interaction rather than a static collection of questions. Tasks, tests, and difficulty can be generated at evaluation time, reducing the opportunity for direct task memorization. The mirrored roles also connect implementation with software review, a closer approximation of collaborative engineering than patch generation alone.

The evidence is still preliminary. The study covers a limited number of repositories, languages, and participants, and the quality of the arena depends on agents consistently creating subtle, valid, and solvable defects. Larger and longer evaluations will be needed, along with safeguards against agents exploiting the protocol itself. Even so, SWE-Duel points toward a useful future for coding benchmarks: an evaluation environment that becomes harder as its participants become better.

Source: Hugging Face Daily Papers