Longtail-SWE Brings Underrepresented Languages to Coding-Agent Evaluation

Introduction

Coding-agent evaluations have largely concentrated on a small set of popular languages and ecosystems. That focus makes benchmark construction easier, but it leaves an important question unanswered: can an agent understand requirements, navigate a repository, and implement a fix when the language has less representation in training data and existing evaluations? Longtail-SWE is designed to address this gap.

What the benchmark provides

• Thirteen underrepresented languages. The collection includes Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift. Together, they span functional programming, scientific computing, mobile development, and other general-purpose ecosystems.
• Real repository-level work. The benchmark contains 202 issue-resolution tasks. Each task is connected to a GitHub issue and its corresponding pull request, with an executable environment and tests. Agents therefore have to interpret a practical requirement and modify an existing codebase rather than solve an isolated exercise.
• Quality control beyond the reference patch. Longtail-SWE uses a solver–judge–repairer pipeline. A solver first attempts a candidate task. A judging stage then examines the task, the solver’s trajectory, and test results. The task description or tests can subsequently be revised when they are ambiguous, reject valid alternative implementations, or fail to reject incorrect ones.
• Multiple evaluation dimensions. The study evaluates nine model configurations and examines not only task-solving performance, but also consistency across repeated attempts and resource usage. An additional set of easier tasks is released to support evaluation with smaller models.
• Reproducible infrastructure. The authors release benchmark instances, executable environments, construction and evaluation tooling, and agent trajectories. This makes it possible to reproduce experiments, inspect task decisions, and build new evaluation methods on top of the collection.

Why it matters

Longtail-SWE is notable not simply because it adds more languages. It connects language diversity with benchmark reliability. In smaller ecosystems, the pool of candidate tasks is limited, so unclear issue descriptions or defective tests can cause valuable engineering problems to be discarded. Tests built too closely around one reference patch can also treat an implementation detail as if it were the requirement. By using solver behavior and execution evidence during review, the project offers a practical way to identify and repair these weaknesses.

For model developers, the benchmark can expose how agents generalize when language conventions, libraries, and toolchains differ from those found in mainstream evaluations. For benchmark designers, repeated-attempt consistency and resource use complement a single pass-rate number. The available trajectories may also help researchers study where agents fail: requirement interpretation, repository navigation, implementation, or test interaction.

The supplied material does not report detailed rankings or overall scores for the nine configurations. Longtail-SWE should therefore be understood as an open evaluation resource and a methodology for improving task quality, rather than as a claim that one model has already won a cross-language competition.

Source: Hugging Face Daily Papers