A Three-Layer Cache for Low-Latency LLM Web Search on CPUs

Why caching the whole search pipeline matters

An answer engine that searches the live web has more work to repeat than the final LLM generation. A follow-up question may require the system to reload conversation history, search for information that is semantically equivalent to an earlier request, and embed URLs that have already appeared in other sessions. OreoLook, an open-source answer engine, focuses on these intermediate costs instead of treating response reuse as the only form of LLM caching.

The three layers

• Session Context Window: Recent conversation turns are kept in Redis for fast access. When the working window overflows, older material is Huffman-compressed and moved into disk archives. A background LRU eviction process migrates idle sessions and rehydrates them when needed, allowing conversations to resume after hours or days under the configured retention policy.
• Semantic Query Cache: Queries are represented as embedding vectors and compared with cosine similarity. Rephrased requests that express a similar intent can therefore match an earlier entry. When the match is valid, the system can skip parts of the search and synthesis pipeline, including an otherwise unnecessary remote LLM invocation.
• URL Embedding Cache: The same pages can be encountered by many users and sessions. Caching embeddings by URL removes repeated embedding computation across those boundaries.

What the reported deployment shows

The evaluated setup used one Intel Cascade Lake server with 8 vCPUs and 32 GB of RAM. It ran 30 Hypercorn worker processes across three containerized replicas. The authors reported an aggregate Redis keyspace hit rate of 89.3%, roughly 0.1 ms Redis read latency, and 1.38 MB of measured Redis memory overhead.

These results require careful interpretation. A Redis keyspace hit only indicates that a requested key was found; it does not prove that an entire user query was served from cache or that end-to-end latency fell by the same proportion. Semantic caching also introduces practical questions around similarity thresholds, freshness, and isolation between users. Reusing a result can be unsafe when live web content has changed, so cache invalidation and evaluation methodology remain important parts of the design.

Why it matters

The architecture offers a pragmatic alternative to placing a large model on local infrastructure. CPU resources handle web search, session management, caching, and embedding operations, while final synthesis is routed to a remote provider. For small teams building conversational search products, this division can reduce redundant browsing, vector generation, and model calls without requiring local accelerators.

Its broader contribution is architectural: caching is applied to context, query intent, and retrieved-page features, not merely to generated text. That makes the system relevant to future work on long-lived agent memory, cache-aware retrieval, and more rigorous measurements of how often a cache truly eliminates downstream computation.

Source: Hugging Face Daily Papers