LOWN Lets Diffusion VLMs Look at Images Only When Needed

Introduction

Diffusion vision-language models (DVLMs) generate text by progressively unmasking tokens over a series of denoising steps. In many implementations, the entire image is processed at every step, even though the image itself has not changed. This creates a major computational bottleneck: on the benchmarks studied in the paper, visual tokens account for 89% to 97% of the model input.

The paper Look Only When Needed proposes LOWN, or “Look Only When Needed,” a decoding strategy that treats image access as an adaptive decision rather than a fixed requirement. Instead of reading the image throughout the full generation process, the model checks whether the current prediction actually needs fresh visual evidence.

How the method works

The starting observation is that removing the image often does not alter the token selected for unmasking. In comparisons with LLaDA-V, image removal changes the token being unmasked in only 22.8% of decoding steps on average, and in as few as 7.9% of steps for some benchmarks.

LOWN turns this observation into a routing rule:

• The model first predicts masked tokens using the text context without processing the image.
• It compares that prediction with the most recent image-conditioned prediction.
• If the two predictions agree, the previous image-conditioned result is treated as certified and reused.
• If they disagree, the model processes the image again and refreshes the prediction.
• The number of tokens whose predictions are certified determines how many tokens can be unmasked in that step.

This is different from visual token pruning. Pruning permanently removes part of the visual representation, while LOWN preserves access to the full image and only skips repeated visual computation when the model’s current behavior indicates that it is unnecessary. The strategy is also adaptive across tasks. Because different benchmarks depend on visual information to different degrees, a fixed schedule for image access would be too rigid.

Results and limitations

Without additional training or a tuned threshold, LOWN retains 99.6% of LLaDA-V’s score across ten benchmarks while consuming 35.3% of its decoding FLOPs on average. The reported average generation-time reduction is 57.4%. The method also generalizes to other LLaDA-based DVLMs and can be combined with visual token pruning, suggesting that temporal skipping and spatial token reduction address complementary sources of overhead.

The results should nevertheless be read as a decoding optimization, not as evidence that visual information is generally unnecessary. When a task requires fine-grained or frequently changing visual evidence, prediction disagreements may occur more often and the savings can decrease. The supplied material also does not break down latency by hardware, batch size, or image resolution, so real deployment gains will depend on the surrounding inference stack.

Why it matters

LOWN’s main contribution is a simple change in the unit of optimization. Rather than asking which visual tokens can be discarded, it asks when the model needs to consult the image again. Prediction agreement becomes a lightweight, model-internal certificate for reusing earlier multimodal computation. This avoids training a separate routing model and keeps the full visual representation available when needed.

For diffusion-based multimodal systems, where the same context is revisited many times, such demand-driven routing could become an important way to reduce redundant work. It also points toward a broader design principle for inference serving: multimodal inputs do not have to be consumed at a uniform rate throughout generation.

Source: Hugging Face Daily Papers