SplatLab: Moving AI Research Agents from Ideas to 3DGS Experiments

Introduction

Most AI research systems have focused on reading papers, proposing hypotheses, or generating implementation sketches. The difficult part begins afterward: reproducing a baseline, keeping experiments running, handling failed ideas, and ensuring that the numbers in a manuscript can be traced back to real computations. SplatLab attempts to connect these steps in one pipeline for 3D Gaussian Splatting, a rapidly evolving computer-vision area.

What the system does

• It starts with a verified baseline. SplatLab accepts a research prompt, a target-paper URL, or no explicit input. Rather than asking an agent to generate an entire project from scratch, it first builds a reproduction backbone intended to match the published baseline. New candidates then mutate plugins around that backbone.
• It selects ideas through measured competition. Candidate methods are organized across eight islands and compared in a cascade-weighted Elo tournament. GPU-measured PSNR provides the main score, while verbal-gradient feedback is used to guide later iterations.
• It treats failure as evidence. A standing falsifier subtracts Elo when a candidate fails stress tests. This makes robustness part of selection instead of allowing a method to advance solely because it achieved a high score in a favorable run.
• It binds claims to runs. Each numerical statement in the generated manuscript must point to a logged training run. The workflow is not supposed to terminate while a claim remains unsupported, addressing a common reproducibility problem in automatically generated research.
• It closes the loop. The reported pipeline covers training, ablations, and manuscript generation on Mip-NeRF 360 and LLFF. The project introduction says it produced 16 manuscripts end to end without human code edits, with an autonomous run taking roughly 24 to 48 hours.

Results and limitations

On PSNR-targeted experiments, the generated manuscripts reportedly outperform their reproduced targets by 0.18 to 3.78 dB. The largest stated gain, 3.78 dB, comes from an astrophotographic nebula-rendering experiment. These figures should be interpreted within the project’s stated settings: the abstract alone does not establish how well the improvements transfer to other datasets, metrics, or tasks. Independent evaluation would need to inspect the released implementation, training records, baseline fidelity, random-seed controls, and complete ablations.

Why it matters

SplatLab’s broader idea is to make autonomous research auditable rather than merely prolific. Candidates begin from a verified implementation, measured outcomes determine competition, failed stress tests affect future selection, and manuscript numbers must have experimental backing. If this pattern transfers to more vision repositories, research agents could evolve from writing assistants into systems that organize and execute substantial portions of the experimental loop.

That does not mean the system automatically discovers general scientific laws. Its search remains shaped by the objective metric, the available plugin space, compute limits, and evaluation design. A PSNR increase also does not by itself prove better perceptual quality, efficiency, or scientific understanding. SplatLab is therefore best viewed as reproducibility-oriented research infrastructure, not a replacement for human judgment.

Source: Hugging Face Daily Papers