Detecting Hallucinations as Spans with Temporal Signal Fusion

Introduction

Hallucinations in large language models do not always appear as one obviously suspicious token. A wrong claim may extend across several tokens, while clues about it are distributed through confidence patterns, semantic relations, and the surrounding text. This paper addresses a weakness in conventional detectors that score tokens independently, often using only one signal.

From isolated tokens to connected spans

The proposed system treats hallucination detection as a sequence-labeling problem. For every generated token, it constructs a 33-dimensional feature vector based on three broad signal groups:

• Text statistics, which capture observable patterns in the generated output;
• Natural Language Inference features, which estimate whether the answer is entailed by external evidence;
• Language-model surprisal, which measures how unexpected tokens or local passages are to a language model.

A bidirectional gated recurrent unit, or BiGRU, processes this feature stream. Unlike an independent logistic-regression classifier, it can use information from both earlier and later tokens. Evidence from a confident position can therefore influence neighboring ambiguous positions, helping the detector identify a larger hallucinated span rather than judging each token in isolation.

The method does not require hidden states, logits, or other internal information from the model that produced the answer. It operates on generated text and external signals, which is particularly relevant when the target model is closed source.

What the experiments show

On the RAGTruth benchmark, the BiGRU achieves an AUC of 0.840 across 10 random seeds. The paper reports an approximately 11-point improvement over an independent logistic-regression baseline, with a Wilcoxon signed-rank test result of p=0.002. A controlled decomposition attributes most of the improvement to temporal ordering and context propagation rather than to a simple increase in model capacity.

The study also compares recurrent, state-space, and attention-based sequence models. Their results repeatedly approach a ceiling around 0.845. This consistency suggests that, under the tested setup, replacing the sequence architecture may matter less than improving the underlying feature set.

Why it matters

The central insight is conceptual as much as technical: hallucination can be a temporally extended event, not merely a collection of independent bad tokens. This framing may help systems produce more coherent spans for review and improve downstream quality-control workflows.

The detector also shows some cross-model robustness. When evaluated on text generated by language models not seen during training, it loses less than 4% AUC according to the supplied summary. Still, the available material does not include full dataset splits, detailed ablations, or broader domain evaluations. Results should therefore not be treated as a universal guarantee. External evidence quality and NLI reliability remain important practical variables.

For RAG pipelines and output auditing, the paper points toward a sensible design principle: combine several observable signals, then let a sequence model integrate them over context.

Source: Hugging Face Daily Papers