AutoIndex Turns Document Indexing into a Learnable Retrieval Optimization Problem

Introduction

Most retrieval and RAG optimization work starts with the retriever: better embedding models, stronger rerankers, or a small set of preprocessing knobs such as chunk size and overlap. AutoIndex: Learning Representation Programs for Retrieval shifts attention to an earlier stage of the pipeline: what exactly gets exposed to the retrieval system as the indexed representation of a document.

The paper’s central claim is simple but important: document representation should not be treated as a one-time preprocessing decision. It can be learned and optimized against retrieval feedback.

Key ideas

• Representation programs instead of static preprocessing: AutoIndex defines executable transformations that map raw documents into the form used by the index. These programs can slice documents, enrich content, normalize text, reweight fields, or reorganize information before retrieval begins.
• The retriever is held constant: In the reported experiments, BM25 is fixed across all tasks. This isolates the effect of changing document representation rather than attributing gains to a new retriever or reranker.
• Validation-guided program search: At each iteration, agents inspect failures produced by the current representation program, synthesize candidate updates, rebuild the index, and keep only updates that improve validation retrieval quality.
• Evaluation on heterogeneous retrieval tasks: AutoIndex is tested on CRUMB, a benchmark spanning eight retrieval tasks. Compared with a static full-document BM25 baseline, the learned programs improve Recall@100 on every task.

Results and impact

The paper reports average gains of +8.4% in Recall@100 and +8.3% in nDCG@10, with the largest task-level improvements reaching +30.5% in Recall@100 and +43.6% in nDCG@10. The noteworthy point is not just the size of the gains, but where they come from: the retrieval model itself is not replaced.

For RAG builders, this highlights a practical blind spot. Search failures are often blamed on embeddings or rerankers, yet the indexed document form may be the real bottleneck. Long documents may need task-specific slicing; headings, metadata, tables, and repeated terminology may need different treatment; and some fields may deserve more weight than others.

AutoIndex does not make indexing free. It requires validation data, repeated index construction, and evaluation of candidate programs. Its practicality will depend on corpus size, update frequency, and task stability. Still, the framework points to a useful direction: moving index design from manual heuristics toward feedback-driven optimization.

Source: Hugging Face Daily Papers