SpanCalib-VLM Brings Calibrated Span Detection to Vision-Language Hallucinations

Why this matters

Large vision-language models can produce fluent descriptions that are only partly supported by an image. Detecting this problem requires more than assigning a single truthfulness label: a useful detector should identify the exact words or phrases that are unsupported and attach a confidence score that reflects how dependable its judgment is. SpanCalib-VLM addresses both requirements in a hybrid design.

A two-part detector

The approach combines models with different strengths:

• A generative VLM: The authors use the fine-tuned Qwen3.5-4B-SHROOM-SFT model to propose hallucinated text spans. Generative models can achieve strong span recall, but they may be overconfident and require relatively expensive inference.
• A multimodal sequence tagger: This component uses XLM-RoBERTa-Large for language processing and incorporates visual information from a SigLIP encoder through cross-attention. It produces deterministic token-level decisions and better-calibrated probabilities, although its conservative behavior can reduce recall.

Instead of averaging the two systems, SpanCalib-VLM uses a procedure called Union-Calibrated Fusion. The generative model first supplies a broad set of candidate spans. The sequence tagger then evaluates those candidates with its calibrated probabilities, allowing the final system to retain candidate coverage while moderating overly confident predictions.

Reading the reported results

On the English evaluation split of the SHROOM-Visions shared task, the ensemble achieved a Pearson calibration correlation of 0.41, an overall IoU of 0.39, and overall detection accuracy of 70.7%. Its clean-response IoU was reported as 0.91. These figures suggest that the system handles responses without hallucinations relatively consistently, while the broader task still leaves room to improve span boundaries and confidence quality in difficult cases.

The metrics capture different aspects of performance. IoU measures how closely predicted spans overlap with annotated spans, whereas calibration correlation concerns whether confidence values track actual correctness. A detector can therefore find many suspicious phrases yet remain unsuitable for automated triage if its scores are systematically too aggressive.

Implications and open questions

The main contribution is architectural rather than a claim that one model solves hallucination detection completely. By assigning broad candidate discovery to a generative model and probability refinement to a discriminative tagger, the paper presents a modular way to combine recall, latency, and calibration. Span-level signals could support risk-aware answer ranking, targeted human review, or more detailed comparisons between LVLMs.

The supplied results come from an English SHROOM-Visions evaluation split. The material does not report behavior across languages, image domains, or model scales, so broader generalization remains an open question. The authors state that model weights and code are publicly available, making the system useful for reproduction and further study.

Source: Hugging Face Daily Papers