AI Faces a Harder Science Test: Predicting Real Wet-Lab Results
AI for Science is moving beyond the question of whether a model can read papers or run software. A more consequential question is whether it can make a useful judgment before an experiment begins. UniPat AI’s Science Prediction Universe, or SPU, addresses that question by hiding real wet-lab outcomes from models and asking them to rank experimental alternatives before comparing those predictions with measured results.
What SPU measures
The benchmark contains 116 tasks. One hundred are derived from 85 published papers, while 16 come from unpublished collaborative experiments. Together they cover chemistry, materials science, biology, and medicine, with 4,207 experiments in total.
These are not ordinary knowledge questions. A model must understand the scientific objective, search for evidence, choose computational methods, install research software, and produce a ranking within an eight-hour run using eight CPU cores, 16 GiB of memory, and one H20 GPU. The original results remain hidden, and evaluation takes place in an isolated environment to reduce contamination from memorized answers.
SPU uses Spearman correlation to compare a model’s ranking with the measured ranking. It also generates thousands of random rankings to determine whether a submission meaningfully beats chance.
Leading models are still inconsistent
The first round tested 12 models in 13 configurations. Claude Opus 5.5 achieved the highest average score, at 0.486, but only around 34% to 52% of tasks across configurations performed above the random threshold. The result highlights a gap between operational ability and scientific judgment: a model may search quickly and run sophisticated software without reliably identifying which evidence should determine the final prediction.
Performance also varied sharply with the measurement target. Models performed well on some electrolyte conductivity tasks, yet predictions for the electrochemical window of the same materials could fall below the random level. General claims that a model is “good at chemistry” therefore say little about its reliability on a particular experimental objective.
More compute did not guarantee better answers
One of SPU’s most revealing findings concerns reasoning effort. For GPT-6 Astra, the highest reasoning setting increased the median runtime by about 4.8 times and the median API cost by about 5.3 times compared with a medium setting. The prediction score, however, barely changed. Execution traces suggested that the model often settled on an answer relatively early, while later computation mainly produced minor revisions.
Tool use showed a similar pattern. More methods, Monte Carlo sampling, or GPU attempts did not necessarily improve the ranking. What matters is whether computation targets the physical quantities that actually control the experimental outcome, and whether new evidence can trigger a correction to an initial assumption.
Why the benchmark matters
SPU turns an elusive scientific quality—research judgment or “taste”—into an observable process. Researchers can examine which factors a model selected, when it encountered conflicting evidence, whether it changed its ranking, and whether additional compute contributed anything useful.
For laboratories, this creates a more practical way to test whether a model is suitable for a specific class of experiments. For model developers, it points to a concrete weakness: models can make their workflows look sophisticated while failing to ask whether the quantity being calculated is the quantity the experiment actually measures.
If SPU continues to combine public tasks with private experiments, progress toward useful scientific agents can be measured through reproducible predictions rather than demonstrations alone.
Source: QbitAI
Comments
Checking sign-in status...
Loading comments...