MMedAgent-Plus Teaches Multimodal Agents to Orchestrate Medical Specialists

Introduction

Medical AI is increasingly moving beyond isolated prediction tasks. A practical workflow may require several analyses: an image is examined, an intermediate finding is passed to another specialist, and the results are then combined into a response. A general-purpose model may be capable of interpreting images or text, but that does not automatically make it reliable at deciding which specialist to call, in what order, or how to preserve the evidence returned by each tool.

MMedAgent-Plus addresses this coordination problem with a multimodal medical agent. A vision-language planner is connected to specialist models through a unified tool interface. The planner selects and executes tools, maintains the sequence of intermediate results, and produces a final answer grounded in those outputs.

Key points

• Broad specialist coverage. The system integrates 18 specialist models spanning 12 medical tasks and eight modalities, including CT, MRI, X-ray, ultrasound, retinal fundus images, retinal OCT, and pathology images.
• Training for orchestration. The authors curate 208,928 instruction-tuning examples from 32 public medical benchmarks. The data covers single-tool and multi-tool use, as well as single-round and multi-round interactions, shifting the focus from direct answering to workflow planning.
• Separate evaluation of agent abilities. Tool calling, response synthesis, and end-to-end execution are evaluated separately. This makes it easier to distinguish a planning failure from an error in the underlying specialist model or in the final language response.
• Strong tool-use results. Tool-call accuracy reaches 91.0% for single-tool multi-round interactions, 98.1% for multi-tool multi-round interactions, and 80.7% for multi-tool single-round interactions. In 350 end-to-end cases spanning seven tasks, the agent invokes the required specialist correctly in 90.6% of cases.
• High-quality synthesis under a controlled condition. When correct specialist outputs are supplied, the system scores 14.4–14.8 out of 15 for faithfulness, completeness, and conciseness over 600 samples. In a medical question-answering evaluation, its final-answer accuracy is 0.98, matching the dedicated specialist and outperforming the strongest direct-answer baseline reported in the study.

Why it matters

The central contribution is not another single-purpose medical model. It is an attempt to organize multiple capable models into a reusable workflow. This design lets specialists retain their domain strengths while assigning decomposition, routing, sequencing, and evidence-based synthesis to the agent layer. In that sense, the work moves the discussion from which model wins an individual benchmark to whether a collection of models can operate as a dependable system.

The reported results should still be interpreted carefully. The strongest synthesis scores assume that the tool outputs are already correct, while the end-to-end evaluation covers a limited set of tasks and cases. Real deployment would also need to test conflicting specialist results, malformed inputs, failed calls, recovery strategies, privacy controls, and whether clinicians can inspect the reasoning path and intermediate evidence.

MMedAgent-Plus therefore represents a clear direction rather than a finished clinical solution. Its architecture suggests that the next step for medical AI may be verifiable and composable workflow execution: a planner coordinates specialized capabilities, while every conclusion remains tied to identifiable tool outputs.

Source: Hugging Face Daily Papers