Chest X-ray multimodal evaluation: clinical indication helps, report text can leak the answer

Lead

Chest radiograph AI is often described as an image classification problem, but many real datasets contain more than pixels. They may include Clinical Indication, Findings, Impression, and multiple images from the same study. These inputs are not created at the same stage of care. The indication is typically available before interpretation, while Findings and Impression are written after the radiologist has reviewed the exam.

The arXiv paper studies this timing issue directly. Using 15,000 ReXGradient-160K studies with two readable images and five CheXbert-derived report observations, the authors compare image-only models, indication-only models, ordinary fixed-order multimodal fusion, random-swap fusion, DeepSets, and SectionGuard-MI. Image features come from a frozen DenseNet-121 encoder, and text features from Bio+ClinicalBERT. Findings and Impression are treated only as post-hoc leakage controls, not as prospectively valid inputs.

Key findings

• Two images help more than one: Under the U-Ones setting, the primary-image baseline reaches a macro AUROC of 0.643, while using two images improves it to 0.694. The result supports the intuition that multi-view chest radiographs carry complementary information.
• Clinical Indication is highly informative: Indication-only modeling reaches 0.749 AUROC, exceeding the image-only baselines. This suggests that the reason for ordering the exam is strongly associated with report-derived observations.
• Prospective fusion improves performance: Ordinary two-image-plus-Indication fusion reaches 0.780 AUROC. SectionGuard-MI reaches 0.783 AUROC and 0.260 AUPRC. Its AUROC gain over ordinary fusion is small and not statistically significant, while its AUPRC improvement is reported as 0.0289 with adjusted p=0.004.
• Permutation-aware designs are competitive: DeepSets has the highest prospective AUROC point estimate at 0.787. Random-swap fusion has the highest prospective AUPRC point estimate at 0.265 and shows better calibration than SectionGuard-MI.
• Report text reveals leakage risk: Full report text alone reaches 0.979 AUROC and 0.836 AUPRC. Even after exact or expanded masking, AUROC remains above 0.973, indicating that post-hoc report text can make report-derived labels circular.

Why it matters

The most important contribution is not a single winning architecture, but a cleaner evaluation frame for medical multimodal AI. Clinical Indication is a legitimate prospective signal: it is available before the final report and reflects the clinical context. Findings and Impression, by contrast, are written after image interpretation and can encode the very labels the model is asked to predict.

This distinction matters for claims about clinical utility. A model that performs well because it has access to report text is not necessarily reading images or reasoning over patient state; it may simply be exploiting the reporting process. The paper also uses patient-cluster bootstrap replicates to estimate public-test uncertainty, reinforcing another evaluation lesson: samples from the same patient should not be treated as fully independent.

For developers of chest X-ray AI, the message is practical. Report the timing of every input, separate prospective features from post-hoc controls, use patient-aware uncertainty estimates, and be cautious when a very high AUROC comes from text fields derived from the same report as the labels.

Source: arXiv