OmniScientist Brings Raw Multimodal Evidence into the AI Scientist Workflow

Introduction

AI scientist systems are increasingly capable of drafting hypotheses, running code, and preparing manuscripts. Yet many of them still operate on text, labels, or features prepared in advance. That limitation matters when the evidence is encoded in spatial arrangements, temporal dynamics, relationships between channels, or details of an experimental procedure. If an agent never sees those signals, its research questions and conclusions are constrained before the workflow begins.

OmniScientist is presented as an attempt to close that gap. Instead of treating perception as a one-time preprocessing step, it gives an AI scientist access to heterogeneous raw evidence throughout the research lifecycle.

Key points

• Raw multimodal input: The system is designed to handle images, signals, audio, video, 3-D structures, trajectories, tables, formulae, and graphs. The emphasis is on direct access to evidence rather than reliance on only precomputed scalar summaries.
• A complete research pipeline: A perception layer works with three autonomous agents dedicated to ideation, experimentation, and writeup. Observations can influence the research question, experimental decisions, and claims in the final manuscript.
• Programmatic safeguards: The pipeline runs idea, rigour, and claim checks in code. These checks are intended to support novelty screening, statistical validity, execution provenance, and numerical traceability.
• Cross-disciplinary evaluation: The reported evaluation covers 36 real-data cases, five discipline families, four families of scientific evidence, and multiple modalities. OmniScientist completed the path from raw data to a compiled manuscript in all 36 cases and obtained a mean overall paper score of 6.3 with the reference reasoning backbone.

Why it matters—and what it does not prove

The notable design choice is not simply the number of supported formats. It is the attempt to keep observations connected to every stage of research. A visual pattern, temporal signal, or structural relationship can potentially change the hypothesis rather than being reduced to a fixed feature before the agent starts reasoning. This makes the system relevant to research settings where evidence is distributed across modalities and where interpretation depends on context.

The work also highlights the importance of auditability. Provenance and numerical traceability can make an automatically produced result easier to inspect, while code-based checks may reduce some common workflow failures. Still, these mechanisms are safeguards, not guarantees of scientific truth.

A 36-for-36 completion result demonstrates that the pipeline can run end to end under the reported evaluation setup. It does not by itself establish that every generated hypothesis is novel, every conclusion is correct, or every manuscript would withstand independent expert review. Open questions include the stability of raw-data perception, transfer across disciplines, reproducibility of generated findings, and the ability of automated checks to detect hidden bias or flawed experimental assumptions.

OmniScientist therefore represents a shift in the AI scientist discussion: from how many workflow steps an agent can automate to how much of the underlying evidence it can actually access, connect, and trace.

Source: Hugging Face Daily Papers