Should Embeddings Be Replaced by LLMs? The Cost Says No

Introduction

The ability of large language models to produce useful representations has raised a tempting question: should conventional text-embedding pipelines be replaced by LLMs? LLMs can interpret intent, context, and multi-step relationships, so their flexibility appears attractive. But flexibility is not the same as operational superiority. The Embedder’s Dilemma evaluates the trade-off directly by measuring quality, cost, and throughput together.

A controlled comparison

The authors compare 10 LLMs from six families with 26 dedicated embedding models ranging from 118 million to 14 billion parameters. The benchmark contains 37 tasks spanning classification, semantic textual similarity, clustering, pair classification, and retrieval. This range matters because a single retrieval score cannot represent the many jobs embeddings perform in production.

The headline result is a near tie in aggregate quality. Gemini 3.1 Pro, the strongest LLM in the study, scores 77.6, while the best embedding model scores 77.2. A gap of 0.4 points is too small to justify assuming that a generative model is automatically the better universal encoder.

The task-level picture is more informative:

• Reasoning-intensive retrieval: LLMs perform better when ranking requires interpreting intent or making more complex judgments.
• Classification: Dedicated embedding models take the lead, making them attractive for repeated, high-volume decisions.
• Clustering, STS, and pair classification: The two model families are broadly comparable.
• Economics and speed: A benchmark pass can cost as much as $154 with an LLM versus about $0.11 with a comparable embedding model, a maximum difference of 1,431 times. The open LLMs tested also process tokens 2.5 to 736 times more slowly on the same GPU.

Reasoning has a measurable price

Reasoning tokens account for 28% to 81% of LLM inference cost in the tested settings. The researchers’ ablation results add an important nuance: reducing reasoning budgets preserves, and sometimes improves, retrieval quality for most models. This suggests that production teams should measure the reasoning actually required by a task instead of paying for maximum deliberation by default.

The quality-cost Pareto frontier reinforces the point. Leading embedding models remain on the frontier, while only one LLM—Gemini 3.1 Pro—joins them. LLMs therefore offer a strong option for a narrower class of difficult retrieval problems, not a blanket replacement for vector encoders.

What this means for RAG systems

The most practical architecture is division of labor. Use embedding models for similarity search, classification, and clustering, where throughput and predictable cost matter. Bring in an LLM when retrieval depends on reasoning over intent, context, or relationships that a fixed representation may not capture well.

The study does not make embeddings obsolete. Instead, it reframes the deployment question: which stage truly needs reasoning, and which stage only needs a fast, stable semantic representation? Teams should validate that answer with end-to-end measurements of latency, quality, and spend before migrating an established pipeline.

Source: Hugging Face Daily Papers