Frozen Models, Evolving Expertise: How FMEE Learns from Medical Deployment

Introduction

Medical AI systems are often frozen once they are deployed, even though clinical evidence, guidelines, and therapies continue to change. Fine-tuning can refresh a model, but it requires access to weights, additional training data, and a suitable training pipeline. Prompt-only or parameter-free approaches are cheaper, yet they can overfit a fixed validation set, rely on weak domain knowledge, or reduce visual experience to text and lose important image details.

The paper Frozen Models, Evolving Expertise introduces FMEE, a model-agnostic framework designed to let frozen language and vision-language models learn from the cases they encounter in deployment.

Key ideas

• Externalize learning instead of changing weights. FMEE leaves the underlying model untouched and updates a set of external expertise resources. This makes the framework applicable to different open-weight and closed-source models.
• Skill captures procedures. Skill is a natural-language description of how to reason, use tools, and handle a task. After each batch of cases, the procedure can be rewritten based on observed performance.
• Knowledge Memory stores supported facts. The memory is intended for reusable information backed by earlier cases or trusted sources such as PubMed. It is not merely a log of previous model answers.
• MMKB preserves visual experience. The multimodal knowledge base stores visual reference cases together with their source answers. When a case is retrieved, the model is guided to compare that image with the current one, preserving the relationship between evidence and visual context.
• Updates face a continual validation test. After a batch, an optimizer proposes changes from scored trajectories. A change is kept only when it helps on new cases without degrading results on earlier cases.

Why it matters

The evaluation covers six benchmarks spanning clinical diagnosis, clinical workflows, medical reasoning, and medical and non-medical visual reasoning. Four open-weight or closed-source base models are included. According to the paper’s summary, FMEE improves medical-task performance by as much as 34.2% over the corresponding base model.

The important contribution is not a claim that a frozen model has been fully retrained. Instead, FMEE offers a middle path between conventional fine-tuning and static prompting. The model parameters remain fixed, while the deployment layer evolves around procedures, facts, and visual examples. This separation may also improve traceability: a system can retain where a memory item or reference case came from.

The approach does not remove the main risks of medical AI. Memory quality, retrieval errors, contaminated cases, and incomplete validation can all affect later decisions. The new-versus-old case check is intended to limit regression, but it cannot replace clinical review or safety evaluation. FMEE is best understood as a systems framework for giving frozen models controlled access to accumulated experience—not as permanent self-training of the model itself.

Source: Hugging Face Daily Papers