SWE-bench Science Shows Why Scientific Code Repair Is More Than Passing Tests

Introduction

In ordinary software, fixing a bug often means restoring the expected behavior of a program. Scientific software is more demanding because the code may itself function as part of an instrument, simulation pipeline, or evidence-generation process. A small implementation error can therefore alter an experiment or the basis of a scientific conclusion. SWE-bench Science is designed around this distinction: it asks not only whether a coding agent can produce a patch, but whether the patch preserves the scientific meaning of the system.

What the benchmark covers

The benchmark contains 119 tasks drawn from 98 GitHub repositories across 20 scientific domains. The tasks are organized into three paradigms:

• Issue-driven tasks, which reproduce and repair reported defects;
• Expert-exploratory tasks, which require agents to investigate problems using domain understanding;
• Engineering-integration tasks, which test whether a fix works across modules, interfaces, and workflows.

This scope goes beyond evaluations that primarily measure whether a visible test turns green. According to the paper, even the strongest reported system, Claude Code with Opus-5 (max), achieves a Pass@1 below 50%. The result suggests that scientific software repair remains far from routine automated debugging.

Four recurring failure mechanisms

The study groups failures into four categories. The first is a lack of scientific knowledge or an incorrect abstraction of the domain problem. An agent may understand the syntax and local control flow while missing the physical, mathematical, or methodological constraint that defines correctness. The second is misguided exploration or surface-level repair: the agent changes the most visible symptom without identifying the underlying cause.

The third is incomplete repair coverage or failed system integration. A local edit may look plausible but leave related code paths, interfaces, or downstream assumptions inconsistent. The fourth is failure to generalize scientific knowledge beyond observed cases. An agent can fit the provided example yet fail on unseen inputs, edge conditions, or different parameter regimes.

The project discussion adds an important evaluation detail. SWE-bench Science uses a separate private test suite that is not exposed to agents. The verifier starts from a fresh workspace, applies the submitted patch, and evaluates behavior-oriented scientific contracts. Public reproductions are therefore diagnostic rather than the final acceptance criterion. Editing a visible assertion or hard-coding the observed output may make a public test pass, but it cannot satisfy the private checks. The project team reports that Qwen3.8-27B passed all 119 public reproductions but reached full private Pass@1 on only 35 tasks.

Scientific guidance can help—or mislead

The benchmark also includes a paired ablation that removes explicit scientific guidance while preserving the repository and executable engineering context. Its conclusion is nuanced: domain knowledge is not automatically beneficial. Well-grounded guidance can constrain the search space, improve average performance, and reduce token use. Poorly aligned guidance can anchor an agent to the wrong interpretation and make repair less effective.

Why it matters

SWE-bench Science moves the evaluation question from “Can an agent generate a runnable patch?” to “Can it maintain scientific semantics while changing a complex codebase?” That shift matters to research institutions, scientific software maintainers, and model developers. Future systems will need stronger support for domain reasoning, integration testing, uncertainty reporting, and generalization beyond the visible example.

Pass@1 is still only one measurement and cannot replace expert review of scientific conclusions. A dependable research coding agent must distinguish between passing a test and preserving the computation’s intended meaning. It should also expose its assumptions and test boundaries instead of optimizing only for a green check.

Source: Hugging Face Daily Papers