FM²: A federated foundation model framework for heterogeneous multimodal medical imaging

Lead

A central tension in medical imaging AI is that stronger foundation models usually benefit from broad, multi-institutional data, while hospitals cannot simply pool sensitive patient data into a single repository. The arXiv paper “FM²: Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging” addresses this gap with a federated framework designed for multimodal medical imaging under privacy constraints.

The paper goes beyond the standard federated-learning assumption that every client owns roughly the same type of data. In real clinical networks, institutions may differ not only in label distributions, but also in which imaging modalities they can provide. Some clients may share modalities but have different disease distributions; others may hold entirely disjoint modalities. FM² treats this imaging modality heterogeneity as a first-class modeling problem.

Key points

• Two heterogeneity regimes: The authors define an Overlapped setting, where clients share some imaging modalities but differ in label distributions, and a Non-overlapped setting, where clients hold fully disjoint modalities. The latter is especially challenging because representation transfer cannot rely on direct modality overlap.
• Medical-domain backbone training: FM² trains the core backbone from scratch to preserve fidelity to medical imaging rather than relying only on natural-image pretrained models. It can also incorporate biomedical pretrained encoders for vision-language alignment.
• Dual Mixture-of-Experts modules: Each client uses a Class-wise MoE for personalized category knowledge and a Domain-wise MoE for shared cross-modality representations. This separates local task specificity from transferable domain structure.
• Heterogeneous Modality Alignment: The proposed HMA regularizer explicitly aligns modality-specific expert parameters. The paper also states convergence and generalization guarantees with an O(1/√T) rate.
• Caption-enhanced learning: Locally retained GPT-4o-generated captions serve as a textual semantic bridge, allowing clients with disjoint modalities to exchange useful representational signals without centralizing raw data. The same idea is extended to federated medical VQA.

Why it matters

The contribution of FM² is not merely another federated optimization recipe. Its more important framing is that medical AI collaboration is structurally messy: hospitals differ in scanners, protocols, modalities, patient populations, and annotation coverage. A method that assumes every participant owns the same data type risks missing the hardest part of real deployment.

By combining expert routing, modality alignment, and text as a semantic intermediary, FM² offers a systematic route toward privacy-preserving multimodal medical foundation models. The paper reports consistent improvements over state-of-the-art federated baselines on its MIMH benchmark and real-world medical VQA datasets, including out-of-modality generalization.

At the same time, practical adoption will depend on factors not fully answered by an abstract: caption quality control, local compute cost, institutional agreements, auditability, and clinical validation. Still, as an ACM MM 2026 main-track paper, FM² points to an important direction: the next frontier in medical foundation models may be less about centralizing more data and more about learning across fragmented, heterogeneous, privacy-bound environments.

Source: arXiv