ReproBridge Makes ML Reproduction Less Dependent on Guesswork

Introduction

Public code does not automatically make a machine learning experiment reproducible. Dependency versions, preprocessing choices, parameter settings, launch order, and fixes for failed runs may be scattered across a paper, a repository, and auxiliary outputs. For coding agents, reproduction therefore involves more than running a script: they must infer the steps that the original authors left implicit.

ReproBridge studies whether those inferences should be preserved for the next agent. Its central idea is straightforward: an agent that has already recovered the missing procedure can leave behind a better handoff than the original repository alone.

What the workflow produces

The project defines a Refined Reproducible Artifact, or RRA, with four main elements:

• a cleaned and repaired codebase;
• documented environment setup;
• experiment launchers that can be invoked directly;
• a reproduction guide explaining settings, rationale, and repairs.

The construction workflow follows “Specify → Execute → Preserve.” An agent first interprets the paper and official repository, then runs and debugs the experiment, and finally records the effective procedure. An independent Reviewer checks the resulting evidence rather than simply trusting a reported number.

ReproBridge-Bench contains 1,029 tasks drawn from 387 experiments across 36 papers. Each task is anchored by five human-verified reference scores, allowing the evaluation to account for variation between independent reproductions. Agents work in fresh sessions and receive either raw code or an RRA, keeping the task fixed while changing the quality of the handoff.

Results and implications

Across four coding agents, refined artifacts increased mean reproduction success from 76.4% to 92.6% and reduced downstream API spending by an average of 19.7%. The spread between models fell from 21.6 to 3.9 percentage points, with every evaluated model exceeding 90% mean success. The ablation study found that removing any component hurt performance, while removing code or environment information caused the largest declines.

The broader lesson is that reproducibility depends partly on preserving execution knowledge, not just publishing files. A repository delivers implementation; an RRA attempts to deliver a validated route through the implementation. Recording why a setting was chosen and how a failure was repaired may be as valuable as releasing the final script.

There are also important evaluation limits. An Auditor traces reported values to output files, execution records, and scripts. Against two human reviewers, it achieved 88.6% precision and 79.5% recall across 818 outcomes. The project also states that its code and data remain under internal review. ReproBridge should therefore be viewed as a promising workflow and benchmark design, with broader independent validation still needed.

Hugging Face Daily Papers