Can AI Agents Actually Close a Physics Research Question?

Introduction

Producing a plausible derivation or calculation is no longer the only question in evaluating scientific AI. A harder test is whether an agent can organize an investigation around a research goal, identify the evidence that is actually needed, validate the result, and stop only when the conclusion is adequately supported. PhysClosureBench, introduced by researchers associated with Shanghai AI Laboratory, is designed to measure that capability.

From local progress to research closure

The benchmark contains 50 expert-verified tasks covering strong-field physics, attosecond physics, and neighboring areas. Each task specifies a research objective and essential scientific constraints, but leaves the investigation strategy open. An agent must decide how to explore the problem, what calculations or analyses to perform, and how to establish that the resulting evidence answers the original question.

The benchmark uses two complementary layers of evaluation:

• Atomic Score measures local scientific accomplishment through 20 criteria, covering pieces of analysis, computation, and evidence handling.
• Research Closure, or RC0–RC4, measures how fully the stated question has been resolved. It focuses on indispensable evidence and the physical conditions under which the conclusion is valid.

This distinction matters. A system can complete several technically convincing steps without connecting them into a defensible answer. Closure requires the agent to align the objective, evidence, limitations, and final claim.

What the results show

Seven frontier language models were evaluated. Claude-Opus-5 achieved the highest mean Atomic Score, 6.29 out of 10, but reached RC4 in only 32.7% of attempts. The gap remains even among high-scoring runs: 36.2% of attempts with an Atomic Score of at least 8 still failed to reach RC4.

The benchmark also reports an important reliability issue. Across three runs, every model reached RC4 on substantially fewer tasks than in at least one of its runs. A successful result therefore does not necessarily indicate a repeatable ability to construct the same evidence chain.

Why it matters

Case analyses identify two recurring failure modes: agents may misinterpret results they generated themselves, or leave the physical applicability limits of a conclusion unresolved. These are not merely formatting problems. They determine whether a result can legitimately support the research claim.

PhysClosureBench shifts scientific-agent evaluation from “does the answer look sophisticated?” to “is there enough evidence to end the investigation?” Future systems will need better evidence planning, result auditing, scope checking, and uncertainty communication if they are to become dependable research collaborators rather than fast producers of plausible intermediate work.

Source: Hugging Face Daily Papers