VAD Separates Visual Evidence from Teacher Bias in Multimodal Distillation

Introduction

Multimodal on-policy distillation is designed to transfer fine-grained visual knowledge from a stronger, privileged-view teacher to a student model. The student first generates its own trajectory, and the teacher then provides next-token corrections along that trajectory. This makes the supervision more aligned with the student’s actual behavior, but it also creates a difficult attribution problem: what part of the correction really comes from the visual evidence?

The paper “VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation” argues that teacher corrections are source-mixed. A correction may reflect image evidence, but it may also reflect linguistic priors or teacher-specific tendencies. If the student learns all of these signals indiscriminately, the distillation process may transfer more than visual understanding.

Key ideas

• The central question is what to distill, not only where to distill. Prior approaches often focus on selecting important positions or weighting visually relevant tokens. VAD asks a sharper question: is a teacher correction actually supported by the image?
• VAD uses a counterfactual comparison. At each prefix generated by the student, the same fixed teacher is evaluated under two conditions: with the relevant visual evidence available and with that evidence removed.
• The output difference defines a visual evidence direction. The change in centered log-probabilities forms a signed proxy that estimates how visual evidence supports or refutes candidate tokens.
• The teacher correction is decomposed. VAD projects the original correction onto the visual proxy, keeping the intervention-aligned component and separating the residual that cannot be explained by the proxy.
• The student learns from a reconstructed target. Instead of using the teacher’s full correction as the main signal, VAD builds a student-anchored target from the visually attributable component. The privileged teacher remains in training only as a weak regularizer.

Why it matters

The method reframes multimodal distillation as an attribution problem. In fine-grained visual tasks, a model may produce a wrong answer not because it lacks language competence, but because it fails to use the right visual details. A teacher can correct the answer, yet that correction may still be partially driven by language priors. VAD tries to isolate the part of the correction that changes when visual evidence is intervened on.

According to the paper, VAD improves over direct privileged-view distillation and visual-advantage weighting across six fine-grained visual benchmarks, tested at both 4B and 9B scales. Token-level and controlled-target analyses suggest that the proxy-aligned component is richer in task-relevant visual corrections and produces stronger target shifts, especially when the evidence refutes an incorrect answer.

The contribution is not a new model release, but a training-signal refinement. For teams building vision-language models, the implication is practical: better distillation may require filtering teacher supervision through evidence attribution, rather than simply imitating a stronger teacher more closely.

Source: Hugging Face Daily Papers