AV-FABLE Brings Dual-Timeline Forensics to Long-Form Video

Introduction

A manipulated long-form video does not have to be fake from beginning to end. A few seconds of altered speech, a replaced sound, or a brief visual edit may be inserted into an otherwise authentic interview, lecture, event recording, or open-scene video. For forensic work, a binary answer about whether the whole video is real is therefore insufficient. Investigators also need the time of the manipulation and the modality responsible for it.

The paper AV-FABLE: Audio-Visual Manipulation Forensics in Long-Form Videos addresses this problem with a benchmark and a dedicated readout architecture. Its central idea is to evaluate audio and visual evidence on separate timelines rather than treating the video as one undifferentiated sequence.

Key points

• A benchmark built for long-form, sparse edits. AV-FABLE provides 16,650 pristine-manipulated twin pairs, covering 998 hours and 27 editing configurations. The data records audio and visual manipulation locations independently.
• Separate timelines improve diagnosis. Joint-timeline evaluation can hide an important error: a model may identify roughly the right moment but fail to tell whether the audio or the image was manipulated. AV-FABLE makes that distinction explicit.
• Prompting alone is not enough. Zero-shot prompting of audio-visual large language models produces near-chance detection and weak temporal localization on the benchmark. This does not necessarily mean that the models lack useful sensory evidence; their language outputs may simply fail to expose it.
• NFSR reads the latent evidence directly. Native Forensics via Separated Readouts freezes the perception features of an audio-visual model and attaches separate temporal decoders for the audio and visual streams. The design is motivated by the observation that forensic signals can be present in representations without being accessible through ordinary answers.
• The method is tested beyond its home distribution. NFSR delivers the best audio-visual localization performance in-domain and across three out-of-distribution settings, including transfer to held-out videos from an external long-video forensic benchmark.

Why it matters

The contribution is not just a larger deepfake dataset. It changes the unit of analysis for long-video forensics. Existing audio-visual benchmarks have often emphasized short talking-head clips, while long-form material introduces changing scenes, background sounds, camera motion, and edits that may affect only a small event. A modality-specific timeline can distinguish detecting an anomaly from explaining where the anomaly comes from.

The results also offer a broader lesson about multimodal foundation models. A model may encode useful visual and acoustic evidence internally while failing to convert that evidence into a reliable natural-language judgment. For tasks that require precise boundaries, a frozen general-purpose perception stack combined with specialized temporal readouts may be more effective than prompt engineering alone.

The benchmark does not by itself solve deployment challenges. Real-world systems will still need to control false positives, handle manipulation types not represented in the benchmark, and present evidence in a form that human investigators can verify. Nevertheless, AV-FABLE provides a clearer evaluation target for those next steps: not merely whether a video is suspicious, but when, how, and in which modality it was altered.

Source: Hugging Face Daily Papers