Nexus Lets Agents Route Tools Without Re-reading Every Schema

Introduction

Tool-using agents built around MCP may have to process a large collection of tool schemas on every turn. As the registry grows, schema prefill consumes more compute and context space, making it a major contributor to time-to-first-token (TTFT). Nexus addresses this bottleneck by separating the question of which tool to use from the more expensive task of encoding every available schema.

How the design works

• Retrieve before generating. Nexus stores tool representations in an INT8 semantic lookaside buffer (SLB). A calibrated cross-encoder margin gate then selects candidate tools, allowing the main context to avoid carrying the complete registry on every request.
• Generate from a compact signature. Arguments are produced over a textual tool signature rather than a full schema KV block. The signature has a median length of about 19 tokens, reducing the amount of schema material placed in the active context.
• Use KV splicing selectively. Nexus also experiments with copying a compiled schema KV block into the live context. This is only a secondary optimization. Rotary positional embeddings make the result sensitive to where the block is inserted: an anchored splice can be output-exact, while an off-anchor placement can distort attention.
• Repair when depth increases. Beyond the threshold P=256, the system applies a depth-adaptive suffix redecode and can escalate to a complete re-prefill. Its “never-regress” property concerns output fidelity, not a guarantee that latency will always improve.

Results and limitations

On Qwen2.5-14B-Instruct Q4_K_M running on Apple-silicon unified memory, retrieval-based routing stays near 89% accuracy as the registry reaches 250 tools. The first argument token arrives 1.66 times sooner than with full-schema re-prefill, while main-context token use falls by roughly 80%. At that scale, a baseline that concatenates every schema overflows the context window.

KV splicing produces a 1.1–1.7x TTFT improvement at moderate context depth, but the benefit narrows toward parity as depth grows. The repair path preserves top-1 agreement with KL divergence close to zero, although latency can briefly fall to 0.98x of the baseline before converging. A reference-free drift gate also fails to predict positional drift reliably, with a reported Spearman correlation of 0.193.

Why it matters

The strongest lesson from Nexus is not that KV caches can be moved freely. It is that tool routing should be decoupled from full-schema prefill whenever possible, while cache splicing should be treated as a bounded optimization with explicit verification and fallback. This could reduce TTFT and context pressure for agents with large tool registries. However, the quantitative envelope is specific to one model, quantization setting, and hardware platform. Broader tests across models, RoPE implementations, context lengths, and realistic tool distributions are still needed.

arXiv