GLIE Compresses Visual Document Retrieval with Generative Embeddings

Introduction

Visual document retrieval must account for more than the words on a page. Layout, images, tables, and local text regions can all affect whether a document matches a query. Late-interaction retrievers preserve this detail by representing each page with many vectors and comparing query vectors against them with MaxSim. The trade-off is substantial index growth: a typical page may require around 1,000 stored vectors.

The paper introduces Generative Late-Interaction Embeddings, or GLIE, as a way to reduce that storage burden without retraining the document encoder. Instead of treating compression as simple deletion or averaging, GLIE learns a compact representation that can also act as a basis for regenerating the original page-level embedding set.

Key points

• The vectors have useful geometry. Across three encoders, the researchers found that the page vectors lie exactly on the unit sphere and cluster near a manifold with an intrinsic dimension of roughly five to six. The observation suggests that the apparent size of the representation overstates the number of independent degrees of freedom.
• Raw k-means centers are poorly calibrated. Standard centroids fall inside the unit sphere even though the original vectors sit on its surface. Using those centroids directly therefore tends to underestimate MaxSim scores. Renormalizing the centroids to the sphere produced a reported gain of up to 0.093 nDCG@5 over raw centroids.
• One compact code serves two purposes. GLIE learns k vectors per page, with k much smaller than the original number. The vectors are used as a lightweight retrieval index and as the input basis for a decoder that reconstructs a fuller embedding set. The document encoder remains frozen; training is limited to a small refiner and decoder.
• Expansion is restricted to likely matches. Search first runs on the compact codes. In the reported evaluation, only the top 20 candidates are expanded before exact MaxSim reranking, keeping the more expensive reconstruction step away from most of the corpus.

Results and implications

On ViDoRe v1, using four vectors per page, GLIE retained nearly 80% of the uncompressed system’s nDCG@5. The best prior post-hoc compression method retained about 70% under the same comparison. The additional network contains 415K parameters, indicating that the approach does not rely on a large auxiliary model.

The broader contribution is conceptual as much as empirical. A compressed multi-vector index does not have to be only a subset of tokens or a collection of local averages. It can instead be a compact coordinate system from which the detailed representation is regenerated when needed. This is especially relevant for systems where storage is constrained but fine-grained late interaction remains valuable.

The reported gains should still be read within the scope of the supplied evaluation. Performance will depend on the encoder, document distribution, number of stored vectors, candidate cutoff, and the cost of decoding. Nevertheless, GLIE offers a practical direction for scaling visual document retrieval: keep a small representation for broad search, then spend computation only on candidates that have a realistic chance of ranking highly.

Source: Hugging Face Daily Papers