VisCo Uses VLMs Themselves to Compress Visual Tokens

Introduction

Vision-language models have become increasingly capable at image understanding, visual question answering, and multimodal reasoning. But that capability often comes with a heavy cost: images are represented as many visual tokens, and these tokens must be processed alongside text during inference. More visual tokens usually mean higher latency, larger memory usage, and more expensive deployment.

The paper “VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression”, featured on Hugging Face Daily Papers, proposes a different way to think about this bottleneck. Rather than attaching a new compression module to a VLM, VisCo asks whether the pretrained model can serve as its own visual compressor.

Key points

• The bottleneck is token volume
  VLMs often pass a large number of visual tokens into the language backbone. This is useful for preserving image detail, but it also increases attention computation and memory consumption. The issue becomes more visible when models handle high-resolution images, multiple images, or long multimodal contexts.

• Existing compression strategies have trade-offs
  Training-free methods are easy to apply, but they usually rely on heuristic scoring or token selection. Under aggressive compression, the paper argues, these methods can suffer significant performance degradation. Training-based approaches can be stronger, but many introduce external compression modules, forcing the VLM backbone to adapt and raising retraining cost.

• VisCo’s core idea: use the model’s own encoding ability
  The authors argue that strong compression depends on strong information encoding. Pretrained VLMs already contain substantial representational ability, but existing compression methods may not fully exploit it. VisCo therefore reuses the VLM itself as an intrinsic encoder instead of treating compression as a separate add-on.

• Memory tokens become compact visual carriers
  VisCo is described as a training-efficient, parameter-sharing autoencoder. During encoding, visual information is compressed into a small set of memory tokens. During decoding, hierarchical information is transferred from the encoding side to help preserve useful visual content. This design tries to reduce token count without discarding the pretrained model’s priors.

• Reported gains are strongest under aggressive compression
  According to the abstract, VisCo outperforms previous methods across all evaluated compression ratios, with larger gains when the compression ratio becomes more aggressive. The authors also report stable behavior even in the extreme single-token setting. Another interesting observation is that learned memory tokens can improve the base model when combined with original visual tokens, suggesting they may contain complementary representations rather than merely compressed leftovers.

Why it matters

VisCo is notable because it shifts visual token compression from external pruning or auxiliary modules toward self-compression. If the approach generalizes across more backbones and tasks, it could become a practical route for reducing multimodal inference cost while preserving the strengths of pretrained VLMs.

For developers, the appeal is clear: fewer visual tokens can mean lower latency and memory use, especially in multi-image or resource-constrained applications. At the same time, the method’s reliance on the model’s own representations may make it less disruptive than adding a separate compressor.

That said, the provided material does not include full benchmark tables, exact cost measurements, or deployment details. A careful reading of the paper and code is still needed to judge engineering complexity and real-world robustness. The Hugging Face page links to a GitHub repository, and an author comment states that the work was accepted by ACM MM 2026.

Source: Hugging Face Daily Papers