Clinical AI Benchmarks Miss How Clinicians Actually Use LLMs

Introduction

Healthcare language models are often judged with exam questions, curated cases, and specialist question-answering datasets. Those tests can reveal useful capabilities, but they may not represent the work clinicians actually delegate to an AI assistant. A study featured in Hugging Face Daily Papers compares real clinical use with public benchmarks and identifies a substantial gap between the two.

What clinicians ask an AI assistant to do

The researchers analyzed 127,833 queries submitted during an eight-month rollout of an institutional assistant. The users included 6,342 physicians, advanced practice providers, and nurses across 35 specialties. Each query was characterized with RCQ-Map, a framework validated by clinicians and designed to record task type, intent, answerability, missing information, and potential harm.

The largest category was documentation and administration, representing 36.2% of queries. Knowledge retrieval accounted for another 28.9%. Together, these two areas made up nearly two-thirds of observed use. Diagnostic requests represented only 3.7%.

That distribution challenges a common mental model of clinical AI. In deployment, the assistant is more often asked to organize information, support routine processes, retrieve knowledge, or help with records than to make a diagnosis. A model that performs well on diagnostic vignettes may therefore be strong on one visible capability while remaining insufficiently tested on much of the work it will receive in practice.

More than a third of queries could not be answered well as posed. This does not necessarily mean that the model lacked the relevant medical knowledge. Some requests may have omitted necessary context, depended on unavailable records, or lacked a clearly defined objective. In healthcare, evaluation should consequently measure whether a system can detect missing information, ask for clarification, and avoid unsafe overconfidence—not merely whether its final answer looks plausible.

The benchmark mismatch

The researchers applied RCQ-Map to 58 public benchmarks drawn from major evaluation suites and frontier-model reports, assembling the results into the Clinical AI Benchmark Atlas. The median benchmark contained no documentation requests at all. Its task distribution shared only 31% of the mix observed in real clinical use.

Notably, benchmarks explicitly designed to resemble clinical practice were not individually closer to real use than benchmarks used in frontier model reports. This does not make exam-style or case-based evaluations useless. They can still test medical knowledge and selected reasoning skills. The problem is treating them as a complete proxy for deployment performance.

Why this matters

The study suggests that clinical AI evaluation should begin with the workflow rather than the model. Hospitals and developers need to ask which roles will use the system, which tasks occur most often, what information is available, and where an incorrect response could cause harm. Evaluation should then combine knowledge tests, case reasoning, realistic documentation and retrieval tasks, clarification behavior, traceability, and safety checks.

Standardized benchmarks will remain valuable, but they should be interpreted as measurements of specific capabilities rather than universal readiness scores. Aligning benchmark composition with actual usage would make model comparisons more meaningful and give healthcare organizations a better basis for deployment decisions.

Source: Hugging Face Daily Papers