HuatuoGPT-3: Adapting Medical LLMs with Reinforcement Learning Alone

Introduction

Turning a general-purpose language model into a medical specialist usually follows a familiar recipe: supervised fine-tuning first, followed by reinforcement learning. This SFT+RL pipeline provides a useful starting point, but it also adds multiple optimization stages and may narrow the model’s exploration space too early. HuatuoGPT-3 investigates a different route: direct medical domain adaptation from a base model using reinforcement learning alone.

Its central method is One-stage Policy Optimization, or OnePO. The key design choice is to treat teacher-generated answers as temporary guidance. They help the policy get started, but are not kept as a permanent behavioral target.

Key points

• A direct alternative to SFT+RL. The paper argues that supervised demonstrations can provide a convenient cold start while encouraging early imitation. Pure on-policy reinforcement learning has the opposite problem: without useful initial behavior, it may struggle to discover informative medical responses.
• Two failure modes are highlighted. In mixed-policy training, important tokens in teacher answers may have very low probability under the current policy. Their learning signal is therefore weak in the early stages, a problem the paper calls “Gradient Starvation.” Later, continuing to train against old teacher answers can hold the policy close to an outdated distribution, creating “Teacher-Distribution Anchoring.”
• Adaptive objectives and teacher retirement. OnePO uses Adaptive Objective Evolution to place more emphasis on informative, low-probability teacher tokens. It also introduces Teacher Retirement: once the current policy can outperform a teacher response, that response is discarded instead of remaining an enduring imitation constraint.
• Reported gains with limited data. In the medical adaptation experiments described in the abstract, OnePO reaches 67.2 on HealthBench Total using only 20,000 training samples. This is reported as 2.7 points above SFT+RL and 7.4 points above pure RL.
• Scaling into an open model family. The resulting HuatuoGPT-3 series is described as open source. Its 27B variant reportedly scores 70.1 on HealthBench Total and 71.4 on HealthBench Professional, exceeding frontier systems including GPT-6 Astra according to the paper abstract.

Why it matters—and what remains open

The broader contribution is a shift in how domain adaptation is framed. Instead of treating supervised data as the indispensable foundation of specialization, HuatuoGPT-3 asks whether reinforcement learning can use teacher behavior selectively, then move beyond it as the policy improves. This could be relevant to other expert domains such as law, finance, and programming, where imitation alone may limit later optimization.

The reported results should nevertheless be read within their stated evaluation setting. Performance on medical benchmarks is not the same as clinical reliability, and the abstract does not establish safety in real-world diagnosis or treatment. Reproducibility across model sizes, domains, reward designs, and deployment conditions will be important for assessing whether OnePO is a broadly useful adaptation recipe or a method whose strongest gains depend on a specific experimental setup.

Source: Hugging Face Daily Papers