CASE Trains Medical Agents to Seek Longitudinal Evidence

Many medical AI benchmarks begin with a carefully selected packet of evidence. The model may be asked to interpret a scan, summarize a record, or answer a diagnosis-related question, but it is rarely evaluated on whether it knows what information is missing. In clinical practice, that distinction matters: relevant clues can be distributed across visits, organs, imaging sequences, and slices.

The CASE project—Clinical Agents for Seeking Evidence—targets this gap by treating medical reasoning as an investigation rather than a one-shot response. Its agents operate inside a patient-specific environment containing clinical context and longitudinal, multi-sequence MRI. Instead of receiving a fixed evidence bundle, an agent can decide which visits and anatomical structures to inspect, which modalities and slices to retrieve, and when to use specialist tools. It then compares the gathered observations before producing a conclusion.

Key points

• A longitudinal multimodal benchmark. Built on UK Biobank data, the benchmark covers 4,739 participants and 50,401 clinical questions derived from real-world ICD-10-coded diagnoses. Each question is linked to clinical information and MRI from baseline and follow-up visits.
• Role-specific, tool-using agents. CASE provides a harness that allows compact vision-language policy models to make selective evidence-gathering decisions. The goal is not to force every case through the same browsing routine, but to test whether the model can choose an appropriate investigation path.
• Post-training for policies, not only answers. Supervised fine-tuning transfers evidence-seeking behavior from 14,734 frontier-model interaction trajectories. Agentic reinforcement learning then uses the learner’s own interactions with its environment. Privileged on-policy self-distillation and rubric-based language-model feedback further target the quality of evidence-to-conclusion reasoning without prescribing a single tool sequence.

The authors report that, under matched evaluation settings, their Qwen3-VL-8B-based agent achieves more than 16% and 10% relative gains in answer accuracy over GPT-5.4 and Claude Opus 4.8, respectively. The broader contribution is not simply a leaderboard comparison. CASE changes what is being measured: a useful medical agent must demonstrate how it searches, compares, and grounds its answer, not merely how fluently it explains information that someone else selected.

This approach could make weaknesses in clinical agents easier to examine. Missing a follow-up visit, choosing an irrelevant sequence, or failing to compare changes over time are different failure modes from misreading a provided image. Making those decisions explicit creates a more informative target for training and evaluation. At the same time, the reported results should be interpreted within the study’s setting. The benchmark is built around a particular data source and longitudinal MRI environment, so transfer to other institutions, diseases, modalities, and clinical workflows remains an open question. CASE is therefore best viewed as a step toward more auditable evidence-seeking agents, rather than as evidence that such systems are ready for unsupervised clinical use.

Source: Hugging Face Daily Papers