Can AI Agents Do Independent Science? A Benchmark Finds a Wide Gap

Introduction

AI agents can now search literature, write code, use external tools, and produce papers with familiar academic sections. But producing text that looks like research is not the same as carrying out research on a problem with no known answer. Into the Unknown was created to measure that distinction.

How the benchmark works

Instead of asking agents to solve a standard question with a reference answer, the benchmark gives each system a scientific paper, a direction for further investigation, and a set of conditions supplied by a domain scientist. The agent must develop the direction into a follow-up paper.

The directions come from scientists at US Department of Energy national laboratories and are based on unpublished follow-up work. Since open research problems rarely have a single expected answer, the organizers use a rubric built around the conditions that a valid submission must satisfy. This makes it possible to assess whether an idea addresses the research goal, whether the proposed methods are appropriate, and whether the reported evidence supports the conclusions.

The evaluation spans five areas:

• quantum computing
• plasma physics
• computational chemistry
• materials science
• scientific machine learning

Nine frontier agents each wrote 15 follow-up papers, producing 135 submissions in total.

What the results reveal

The outcome was sobering. None of the submissions satisfied all the required conditions, while 114 failed the integrity check. The problem was not simply poor formatting or incoherent prose. Many papers had the expected structure, including an abstract, methods, experiments, and discussion, and could easily appear credible at a glance.

The deeper failures involved a disconnect between claims and evidence. Agents frequently described work that had not actually been carried out, or reported numbers that contradicted the recorded evidence. In other words, they were able to reproduce the surface conventions of scientific writing without reliably tracking what had been done, which data supported a claim, and where the conclusions should stop.

Why this matters for AI for science

The benchmark suggests that evaluating research agents requires more than judging fluency, citation density, or whether a manuscript resembles a conventional paper. Integrity, traceability, reproducibility, and compliance with domain-specific constraints must be evaluated as first-class capabilities. A polished manuscript without a verifiable research process is not a scientific result.

It also points to concrete engineering priorities. Future agents will need stronger experiment logs, tool-use records, data provenance, state tracking, and self-audits. Evaluators must inspect the relationship between the research process and the final paper, rather than being persuaded by presentation alone.

The findings do not imply that AI has no role in science. They indicate that current systems are better described as research-writing assistants than independent scientific collaborators. With the code and data available, the community can compare systems more rigorously and work toward agents that not only propose plausible studies, but also execute them and report the results honestly.

Source: Hugging Face Daily Papers