Model Routing Is Not Just Model Selection: IBM Research Reframes the Enterprise Agent Trade-off

Introduction

Model routing sounds like an obvious win for enterprise AI. Send simple requests to cheaper models, reserve premium models for harder tasks, and route specialized workloads to models that fit them best. In practice, IBM Research argues in a Hugging Face Blog post, the problem quickly stops being about choosing a model and becomes about optimizing an entire system.

Key points

1. Cost is not the same as published model pricing
   The post highlights results from 417 tasks on the AppWorld Test Challenge using the same CodeAct agent. Claude Sonnet 4.6 cost $79 in total, or about $0.19 per task, while GPT-4.1 cost $155, or about $0.37 per task. This is counterintuitive because GPT-4.1 has lower token prices on paper, and Sonnet needed roughly three times as many reasoning steps. The missing variable was caching. Agent workloads often reuse large parts of context across multiple steps, and cache-read pricing can sharply change the effective cost of a model.

2. Complexity is not visible task difficulty
   A difficulty-based router assumes the system can identify hard and easy tasks before execution. But a prompt such as “summarize this contract” may trigger retrieval, compliance checks, tool use, and several rounds of refinement. Meanwhile, a technical-looking prompt may be handled efficiently by a smaller specialized model. In enterprise settings, routing also has to respect data residency, privacy rules, approved model lists, and other governance constraints.

3. Latency is not just model speed
   End-user latency depends on more than model size. Routing itself adds overhead. Hardware placement, endpoint load, warm or cold caches, and serving conditions may dominate response time. Routing once per task has limited overhead, while routing at every agent step can improve adaptivity but increases operational complexity and latency risk.

4. Routing should be treated as multi-objective optimization
   IBM Research says it moved away from asking which model is best for a task and instead optimized cost, quality, and latency together. In its AppWorld experiment, different router configurations formed a cost-accuracy frontier. One latency-optimized configuration reached 84% accuracy at $93 and 83 seconds, reducing cost by 21% and latency by 9% compared with running Opus alone, with a 4% accuracy drop.

Why it matters

The broader lesson is that routing is not primarily about model selection. Models are one variable among caching behavior, infrastructure state, workload patterns, reliability, and compliance requirements. For enterprise agents, the best router is not the one that always picks the strongest model, but the one that finds the best operating point for the whole system.

Source: Hugging Face Blog