SWE-Bench ProMax Raises the Bar for Coding Agents with Multilingual Refactoring Tasks

Introduction

AI coding agents are moving beyond code completion and isolated bug fixes into longer, more complex software engineering workflows. Yet the benchmarks used to measure them are under pressure. The paper behind SWE-Bench ProMax argues that existing evaluations are both saturating and facing credibility issues. In particular, a recent audit cited by the authors found that many unsolved SWE-bench Verified instances contain flawed tests: some are too narrow and reject valid fixes, while others are too broad and check requirements that were never stated.

SWE-Bench ProMax responds by focusing on a harder and more realistic task family: large-scale code refactoring.

Key points

• Refactoring instead of simple repair: The benchmark evaluates whether agents can make coordinated, behavior-preserving changes across a codebase, not just patch a local defect.
• Multilingual coverage: The 170 instances span Python, Java, TypeScript, Go, C, C++, and Rust, reducing dependence on a single programming ecosystem.
• Grounded in real commits: Tasks are drawn from real software changes, making them closer to practical maintenance work than synthetic exercises.
• Stronger curation process: The authors describe a multi-stage expert review pipeline. Issue descriptions are rewritten from scratch to make requirements precise, and test suites are manually checked to remove tests that are either too narrow or too broad.
• Larger engineering scope: Each instance averages 11.4 modified files and 261.6 lines of code, emphasizing repository-level understanding and cross-file coordination.

Why it matters

The benchmark is important because real software engineering rarely consists only of fixing one failing assertion. High-value maintenance often involves reorganizing code, migrating APIs, simplifying duplicated logic, or adjusting module boundaries while preserving behavior. These tasks require a model to understand intent, dependencies, and project structure over a longer horizon.

SWE-Bench ProMax also highlights a growing concern in AI evaluation: test quality can shape what models appear to be good at. If the tests are ambiguous, overfitted, or misaligned with the stated task, leaderboard performance becomes less meaningful. By rewriting task specifications and manually reviewing tests, the benchmark attempts to make the scoring signal more trustworthy.

The available material does not provide a full model leaderboard or detailed performance comparison, so the main contribution to emphasize is the benchmark design itself. As coding agents are increasingly expected to operate inside real repositories, evaluations like ProMax may better separate systems that can write plausible code snippets from those that can safely refactor software at project scale.

Source: Hugging Face Daily Papers