SWEeper-Bench Tests Whether Coding Agents Can Find Interactive Bugs

Introduction

Coding agents are increasingly expected to deliver usable interactive software, not merely produce code in response to a prompt. That shift exposes a weakness in conventional software-engineering evaluations. Many benchmarks tell an agent which bug to fix, so they measure patch generation after diagnosis has already been completed. They do not adequately test whether an agent can discover a failure that nobody has reported.

SWEeper-Bench targets this missing part of the development loop. It asks an agent to explore an application, identify an interaction-dependent defect, and repair it without being given the exact failure description.

What the benchmark measures

• 200 real-world tasks: The tasks are built from real bug fixes in 200 widely used open-source web applications.
• Open-ended requests: Instead of a precise issue report, an agent receives the codebase and a broad request, such as finding and fixing problems in Focalboard’s board sharing and membership features.
• Interactive investigation: Agents must launch the software and construct user-like action sequences. Reading source code alone is often insufficient because the failure may emerge only after several state changes.
• Browser-based verification: An agentic verifier evaluates the submitted repair according to what a user can observe and do in the browser.
• A substantial performance gap: Among 15 frontier agents, the strongest system passed 59.0% of the tasks.

The central bottleneck

The benchmark’s trajectory analysis points to bug identification as the hardest stage. Agents frequently reach the relevant feature but stop at shallow checks: a page loads, a control responds, or a normal path appears to work. They may never execute the particular sequence that exposes the defect. In other cases, they encounter an unexpected result but fail to recognize it as a violation of the feature’s intended behavior.

This distinction matters. Interactive bugs are not always visible in isolated functions or static code patterns. They can depend on application state, navigation order, permissions, or the relationship between several UI actions. Effective debugging therefore requires more than code search and patch synthesis. It requires planning experiments, tracking state, interpreting observations, and deciding which hypothesis to test next.

Why it matters

SWEeper-Bench moves coding-agent evaluation closer to production reality. In real software development, users should not have to serve as the first line of quality assurance for every hidden interaction failure. An agent that can only repair a clearly described issue automates part of the loop; an agent that can discover and verify the issue can potentially close the loop itself.

For researchers, the benchmark separates known-bug fixing from open-ended defect discovery. For tool builders, its results highlight the need for better browser control, exploratory test generation, anomaly recognition, state tracking, and regression verification. The 59.0% ceiling also shows that autonomous software sweeping remains unsolved. As agents generate more interactive applications, their ability to test those applications like users may become as important as their ability to write the initial implementation.

Source: Hugging Face Daily Papers