Safe-Psych Tests Whether LLMs Know When Not to Diagnose

Introduction

Most medical AI benchmarks ask models to answer after all relevant information has already been provided. Real clinical work is messier. A patient’s history may unfold gradually, key symptoms may need follow-up questions, and the right decision at an early stage may be to wait rather than diagnose. The arXiv paper “Ask Before You Diagnose” addresses this gap with Safe-Psych, a benchmark designed to test whether large language models can handle diagnostic uncertainty in psychiatry.

Key points

• The benchmark evaluates timing, not just final accuracy. Safe-Psych asks whether a model should diagnose, request clarification, or abstain at each stage of a case.
• It uses real psychiatric clinical notes. The dataset contains more than 1,000 real-world psychiatric notes, segmented to simulate evidence becoming available over time.
• Psychiatrists define the expected action. Each stage is assigned one of three labels: DIAGNOSE, CLARIFY, or ABSTAIN.
• Strong models still struggle with calibration. The paper reports that under-abstention exceeds 60% for most evaluated models, meaning they often provide answers when the evidence is insufficient.
• Safety prompting shifts the error pattern. Safety-aware prompts reduce premature commitment, but the authors find that this can move models toward excessive abstention instead of reliable clarification.

Why it matters

Safe-Psych highlights a limitation in how medical LLMs are commonly evaluated. A model that performs well with complete information may still behave unsafely when a case is incomplete. In psychiatry, that distinction is especially important because diagnosis often depends on timelines, context, comorbidities, and information that may not be available at the first turn.

The paper’s results also challenge the assumption that stronger general capabilities automatically produce better clinical judgment. In the sequential setting, models frequently diagnose before enough evidence is available and rarely ask for clarification unless explicitly prompted. Those premature diagnoses are less accurate than diagnoses made at the appropriate stage.

For developers, Safe-Psych suggests that medical AI systems need training and evaluation around uncertainty management: knowing when to ask, when to wait, and when not to answer. For deployers and regulators, it points to a broader safety requirement. Clinical AI should not be judged only by whether its final answer is correct, but also by whether it reaches that answer responsibly.

Source: arXiv