AI Research Agents Can Tune Models—But Can They Do Science?

Introduction

Calling a coding agent an automated researcher implies more than the ability to write training code. Scientific work requires identifying meaningful questions, understanding domain constraints, choosing informative experiments, and deciding when an apparent improvement is actually significant. Scientific machine learning is a useful stress test because progress often depends on more than routine hyperparameter tuning.

Agent4Sci places frontier agents in a long-horizon setting and asks whether they can improve the published best models on scientific machine learning benchmarks. The study covers 20 benchmarks spanning 11 domains and emphasizes the nature of the progress agents make rather than presenting a conventional leaderboard.

Key findings

• Frontier agents improved on the published best result on every benchmark included in the evaluation.
• Claude Opus achieved a 10.4% mean relative improvement, while Codex reached 4.4% in the reported baseline setting.
• Most gains came from familiar machine learning engineering moves, including changes to training, architectures, and data handling, rather than from new scientific hypotheses.
• Trajectory analysis identified three recurring weaknesses: poor calibration of novelty, underuse of available resources, and a rapid shift into narrow local sweeps.
• Adding lightweight skills for research ideation, experiment management, and scheduled checklists raised Claude Opus to 20.1% and Codex to 8.1%, while producing more varied and domain-relevant interventions.

Why it matters

The paper’s central contribution is to separate “improving a benchmark” from “doing science.” An agent may outperform an existing model by searching a large space of engineering choices without understanding why those choices should work. That capability is useful, but it is not equivalent to generating a new explanation, identifying a hidden variable, or designing an experiment that changes the direction of inquiry.

The results also suggest that better research workflows may be as important as larger models. Structured ideation, experiment records, resource planning, and regular reviews help prevent agents from collapsing into repetitive local optimization. Yet these additions should not be mistaken for domain expertise: they improve the process, but do not guarantee that the resulting ideas are scientifically novel or correct.

For future evaluations, raw score improvements should therefore be paired with measures of reproducibility, interpretability, transfer to unfamiliar tasks, and the quality of the reasoning behind an intervention. Agent4Sci points toward a more demanding standard for scientific agents: moving from automated ML engineering toward systems that can propose, test, and revise meaningful scientific ideas.

Source: Hugging Face Daily Papers