Right Anatomy, Wrong Timing: Auditing Explanations for Echo-Based EF Models

Lead

In medical AI, saliency maps are often used as a quick test of trust: if a model appears to look at the right anatomical region, its prediction is considered more credible. For echocardiography-based estimation of left-ventricular ejection fraction (EF), deep video models can reach near-expert performance, and post-hoc explanations such as Grad-CAM or transformer relevance maps are increasingly used to support claims of clinical plausibility.

This arXiv paper challenges that shortcut. EF is not defined by anatomy alone. It depends on two key moments in the cardiac cycle: end-diastole (ED) and end-systole (ES). A faithful explanation for an EF model therefore needs to be correct in both space and time.

Key points

• The authors fine-tune two different EF regressors on EchoNet-Dynamic: a self-supervised VideoMAE transformer and a Kinetics-pretrained R(2+1)D convolutional network.
• They use architecture-matched attribution methods: Chefer relevance for the transformer and Grad-CAM for the CNN.
• The audit measures three aspects: overlap between relevance and left-ventricular masks, deletion AUC, and temporal localization on ED/ES frames.
• Spatially, the models look plausible. VideoMAE reaches an intersection-over-relevance score 2.91 times above chance, while R(2+1)D reaches 1.98 times above chance.
• Temporally, the story changes. The temporal localization index is essentially at chance level, around 0.97 to 1.00, and no better than random attribution.
• A tubelet-occlusion probe suggests the issue is not merely a failure of the explanation tools. The models themselves do not preferentially rely on ED/ES frames, with performance around 0.90 times chance in that probe.

Why it matters

The main lesson is that spatial faithfulness does not guarantee temporal faithfulness. For static imaging tasks, verifying that a heatmap covers the relevant organ or lesion may be informative. But video diagnosis is different: the timing of evidence can be as important as its location.

In EF estimation, a model may consistently attend to the left ventricle while still failing to base its decision on the frames that clinicians would consider decisive. That creates a risk for explainable-AI validation: a visually convincing heatmap can hide a temporally weak or clinically misaligned decision process.

The paper does not argue that Grad-CAM or transformer attribution is useless. Instead, it shows that current audits are incomplete when they focus only on anatomy. For medical video AI, future evaluation may need temporally aware metrics, targeted occlusion tests, and training objectives that encourage models to use clinically meaningful frames.

For developers and clinical validators, the message is clear: “looking at the right place” is not enough. A trustworthy video model must also look at the right moment.

Source: arXiv