CF-Net Frames Hesitation Detection as a Multimodal Conflict Problem

Lead

Recognizing hesitation or ambivalence in video is different from detecting more prototypical emotions such as happiness or anger. The signal is often weak, context-dependent, and expressed through mismatches: a speaker may sound uncertain while using confident words, or maintain a neutral face while the transcript suggests reservation. The arXiv paper CF-Net: Conflict Fusion with Speaker Normalisation and Certainty Weighting for Ambivalence/Hesitancy Recognition proposes a model designed around exactly this problem.

The work was submitted to the 3rd Edition of the AH Video Recognition Challenge at ABAW 11th, ECCV 2026, using the BAH dataset. Its central idea is that ambivalence and hesitancy are not always visible as a single clear expression; they may emerge from tension among visual, acoustic, and linguistic streams.

Key points

• Three frozen modality encoders: CF-Net represents video frames, audio, and transcripts using frozen SigLIP2, HuBERT, and DistilBERT backbones. This lets the system rely on strong pretrained representations while keeping the challenge model focused on fusion and task-specific learning.
• Speaker normalization to reduce leakage: The paper applies per-speaker normalization to backbone features. This is meant to reduce the risk that the model learns identity-specific shortcuts, such as a particular person’s voice, appearance, or speaking habit, instead of the actual AH signal.
• ConflictFusion as the core fusion mechanism: Rather than simply concatenating multimodal features, CF-Net explicitly computes pairwise incongruence between modalities. For example, confident text paired with hesitant speech, or mismatched facial and linguistic cues, may be informative for recognizing ambivalence.
• Training designed for ambiguity: The model combines certainty-weighted focal loss, manifold mixup, and modality dropout. It also adds an auxiliary certainty-regression head that uses ambiguity annotations, aiming to stabilize learning on samples that are genuinely borderline.
• Reported challenge performance: According to the paper, CF-Net reaches a Macro F1 of 0.7155 on the BAH validation set and 0.7364 on the private challenge test set, with AP of 0.7492.

Why it matters

The interesting part of CF-Net is not merely that it uses multiple modalities. Many video understanding systems do that. Its more specific contribution is to treat cross-modal conflict as a first-class signal. For ambivalence and hesitation, this framing is natural: uncertainty can appear in the gap between what someone says, how they say it, and what their face or body communicates.

The use of speaker normalization also addresses a practical issue in affective computing. If a model overfits to identity-related patterns, its apparent performance may not reflect real understanding. Similarly, certainty weighting acknowledges that some samples are not just noisy but inherently ambiguous, which is especially relevant for human social behavior.

Still, the available material is based on the abstract and arXiv entry. It does not provide enough detail here to assess all ablations, dataset biases, or broader comparisons. For now, CF-Net is best read as a targeted multimodal design for a specific AH recognition benchmark, and as a useful example of moving multimodal fusion from feature aggregation toward modeling relationships and contradictions.

Source: arXiv