FrontierCS 2 Tests Whether AI Agents Can Do Computer Science R&D

Introduction

A coding agent that solves a self-contained programming problem is not necessarily capable of computer science research. Research and systems work usually require understanding an existing repository, identifying a promising change, running experiments, and maintaining correctness across different metrics and workloads. FrontierCS 2 is designed to measure that broader capability.

How the benchmark works

The benchmark starts from a published computer science paper and the code released by its authors. The paper’s contribution is removed, and the corresponding deleted code serves as the human reference that defines the target. This gives each task a concrete research-level objective rather than an artificial prompt written solely for evaluation.

Agents are placed inside a working repository. They may inspect the code, run experiments, and submit a patch at any point. After each submission, a development suite returns a score, providing useful but incomplete feedback. The evaluator, the held-out final suite, and the reference implementation remain private. The final submission is judged against that hidden setup, which makes it harder to optimize only for publicly visible tests.

The 48 public tasks cover 13 domains, including machine-learning systems, high-performance computing, databases, security, and robotics. The organizers also maintain a private collection of more than 100 additional tasks for future release.

What the results show

FrontierCS 2 evaluates 12 agents. Claude Opus 5.5 records the strongest result, passing 58% of its runs. That headline number should not be read as evidence of broad reliability: no run passes on 16 tasks. The benchmark therefore exposes a wide gap between solving selected research problems and handling an arbitrary task consistently.

The evaluation also distinguishes a higher aggregate score from an actual pass. An agent can outperform the human reference on a combined final score while still failing one metric or workload. This matters because research systems are often judged by several constraints at once. Better average performance may coexist with a correctness regression, a resource problem, or a failure on a particular workload.

Why it matters

FrontierCS 2 changes the unit of evaluation from a generated program to a sequence of research decisions. Agents must decide what to inspect, which experiment to run, how to interpret partial feedback, and when a patch is robust enough to submit. Those demands are closer to engineering and research practice than one-shot code generation.

The results point to a mixed picture. Today’s agents can reach the level of the human reference on some tasks, but their capability is uneven and far from dependable across the benchmark. On the shared capability scale, the median task is roughly as difficult as what the strongest agent can handle, leaving little margin for reliably harder work.

For model builders, this suggests that progress will require more than better code completion: long-horizon planning, experiment control, debugging, and multi-objective optimization are equally important. For users, a strong score on an isolated task should not be treated as proof that an agent can independently conduct frontier research.

Source: Hugging Face Daily Papers