BioEVAL Tests AI Models on Experimental Reasoning in Bioengineering

Introduction

Many evaluations of large language models in biomedicine still focus on factual recall or exam-style questions. Those tests can indicate whether a model has encountered a concept, but they reveal less about whether it can connect evidence across papers, reason about an experiment, or interpret an image generated in a laboratory setting. BioEVAL, described in a new arXiv paper, is designed to examine those broader capabilities in bioengineering.

Key points

• A multi-institutional scope. BioEVAL was built by 22 research groups and covers 11 major bioengineering subfields, with the authors describing it as a PhD-level benchmark.
• Three different task families. The collection began with 380 multiple-choice questions, of which 359 were retained after auditing. It also includes 218 literature synthesis tasks and 10 multimodal problems involving experimental image interpretation.
• A second layer of quality control. Items were reviewed by their authoring groups and then checked centrally. After model evaluation, a blinded cross-group audit of the highest- and lowest-accuracy multiple-choice items flagged 21 questions for revision or removal. Those items were withheld from the reported results.
• Cloud and locally deployable models. The study evaluates cloud-scale systems such as ChatGPT, Gemini, and Grok, alongside models that can be run on consumer-grade GPUs.
• Uneven performance across tasks. The strongest reported results reached 90% accuracy on the retained multiple-choice set, a 0.72 similarity score for literature synthesis, and 80% accuracy on the small multimodal sample. Performance varied substantially by bioengineering subfield.

Why it matters

BioEVAL is useful because it treats scientific capability as more than a single score. Its task design moves from recognizing technical knowledge to organizing information from the literature and interpreting experimental evidence. That makes it easier to ask a more meaningful question: can a model support parts of a bioengineering workflow, rather than merely produce a plausible answer to a familiar question?

The results should nevertheless be read carefully. A 90% multiple-choice score does not establish that a system can design or troubleshoot a real experiment. Literature synthesis is summarized with a similarity metric, which may not fully capture factual accuracy, source attribution, evidence weighting, or causal reasoning. The multimodal result is also based on only 10 problems, so the 80% figure is better treated as an initial signal than as a stable estimate of visual-scientific competence.

For model developers, the benchmark points to several priorities. Systems need to become more reliable when moving between bioengineering specialties, better at using experimental context, and more capable of grounding literature summaries in verifiable evidence. They also need to connect visual observations with scientific conclusions instead of treating image questions as another form of text classification.

BioEVAL is intended to remain extensible. Its standardized protocols allow additional experts to contribute items and enable future model evaluations under a common framework. If the task pool grows while preserving the current audit process, the benchmark could become a useful reference for measuring whether AI systems are progressing from biomedical question answering toward practical research assistance.

arXiv