SWE-at-Scale Turns Real Bug Fixes into Scalable Coding Tests

Introduction

As coding agents become more capable, evaluation needs to grow beyond a small collection of manually curated issues. Real-world bugs are valuable because they emerge from authentic development conditions, but their natural occurrence is limited. Manual task construction is also expensive. A different line of work asks large models to invent issues with prompts, yet such tasks can miss the complex mechanisms that make software failures realistic.

SWE-at-Scale, proposed by a team from Shanghai Jiao Tong University, takes a reverse-engineering approach. Rather than asking a model to fabricate a bug from general programming knowledge, it studies how real bugs were fixed and turns that repair knowledge into a signal for inducing new failures.

The central idea: invert the repair

A historical patch often encodes more than the final code difference. It can reveal how a defect arose, which conditions exposed it, and what kind of change removed it. SWE-at-Scale builds on the idea that repair supervision can be reversed: knowledge useful for learning how to correct a program can also support learning how to introduce a plausible defect.

The framework can be summarized as follows:

• collect historical fixes from real software projects;
• extract patterns associated with bug formation and repair;
• invert the repair operation into bug-induction supervision;
• transfer those patterns to new code contexts;
• execute and verify the generated results as SWE tasks;
• sample repeatedly to obtain different bug typologies from the same context.

This design aims to preserve the causal and contextual properties of software failures instead of producing errors that merely look syntactically plausible.

Diversity from one code context

One notable claim in the paper is that a single file can serve as the source for multiple distinct bug types. The reported experiments also indicate that diversity continues to scale as generation is repeated, including up to 200 generations per file. For coding-agent evaluation, this matters because an agent should not succeed by recognizing one recurring bug template. It should handle different triggers, symptoms, and repair paths within related code.

The abstract reports that SWE-at-Scale produces tasks that are more diverse and realistic than existing synthesis baselines and that these tasks pose greater challenges to state-of-the-art coding agents. The supplied material does not include benchmark names, exact scores, or dataset sizes, so these should be read as high-level experimental conclusions rather than a detailed ranking.

Implications and open questions

The framework’s broader contribution is to turn a finite collection of real fixes into reusable supervision for task generation. Researchers can use this resource to expand stress tests for coding agents, while benchmark builders gain an alternative to prompt-only issue synthesis.

There are still important conditions behind the approach. The quality of generated tasks depends on the quality of historical fixes, whether the injected defect is valid, and whether the resulting issue can be executed and verified reliably. Future work will need to address difficulty control, repeated patterns, and the risk that agents exploit artifacts of the synthesis process rather than solving the underlying software problem.

Overall, SWE-at-Scale connects bug repair with the controlled creation of testable bugs, offering a data-grounded route toward larger and more demanding evaluations of coding agents.

Source: Hugging Face Daily Papers