RealSWE: How Real User Requests Change Coding-Agent Evaluation

Introduction

Are coding agents solving the kinds of problems users actually submit, or are they mainly handling carefully prepared benchmark tickets? RealSWE examines this question by comparing everyday software requests with tasks from the SWE-bench family. Its central observation is straightforward: benchmark problems are often long, formal, and well structured, while real requests are frequently short, casual, and missing important context.

What the study found

The authors define a six-category taxonomy for information content and analyze prompts from SWE-chat alongside problem statements from SWE-bench Verified and Pro. They also describe linguistic style along four dimensions. The comparison reveals a substantial mismatch:

• Requests containing only a problem statement, either alone or with limited context, make up 88% of real prompts but only 7% of benchmark problems.
• Casual writing appears in 87% of real prompts, whereas 94% of benchmark problems use a formal style.
• RealSWE contains 381 multi-variant task families. Within each family, the underlying task and gold patch remain the same; only the information composition and linguistic style change.
• When seven contemporary LLMs were evaluated, realistic inputs reduced resolution rates by an average of 6.4 percentage points and could change model rankings.

Which details help?

The controlled experiments offer a more precise message than the usual advice to “add more context.” Explicitly describing the Desired Behavior—what the software should do after the fix—and the Motivation for the change significantly affects model performance. By contrast, Environment Information and Reproduction Steps add tokens but showed no measurable benefit in the reported analysis. Linguistic style had comparatively small effects, and those effects varied by model.

For users, the practical lesson is not that every request must become a formal engineering ticket. Rather, a short prompt should make two things clear whenever possible: the intended outcome and the reason the change matters. For agent builders, the results warn against treating strong performance on polished, information-rich tasks as proof that an agent is ready for ordinary developer workflows.

Why it matters

RealSWE is useful because it separates prompt conditions from the underlying coding task. By testing multiple input variants against the same task and gold patch, the framework can study whether a model succeeds because it genuinely handles ambiguity or because the benchmark supplies unusually helpful instructions. This makes the benchmark relevant to prompt design, agent interfaces, and future evaluation methodology.

The findings should still be interpreted within the study’s scope. The task families are derived from SWE-bench Verified and Pro, and the analysis focuses on information composition and linguistic style rather than every aspect of software engineering work. Even so, the broader implication is clear: coding-agent evaluation needs to include the way people actually ask for help, not only the way benchmark authors package a problem.

Source: Hugging Face Daily Papers