Why One Self-Hosted Model Can Serve a Corporate Request Mix

Introduction

For enterprises, deploying an LLM is also a fleet-management problem. Data-residency requirements may rule out external hosting, while successive model upgrades often leave older versions running for compatibility or risk-management reasons. The result is a fragmented serving fleet competing for a finite GPU pool. The work behind this paper takes a consolidation-first approach: route the request mix from more than 200 internal applications to one self-hosted model, then use production failures to guide post-training.

The recipe

• Start with production errors. Instead of optimizing only broad public benchmarks, the team identifies three gaps that matter to its traffic: instruction following, function calling, and the distribution of internal tasks. Offline evaluations are stratified to resemble production traffic.
• Train experts separately. Each axis receives its own GRPO expert. The rationale is that a single joint reward can create interference: improving tool use may alter conversational behavior, while optimizing internal tasks may hurt instruction adherence.
• Treat reward hacking as domain-specific. The reported failure modes differ by axis. Instruction training can lead to semantic collapse, function-calling training can produce over-calling, and internal-task optimization can encourage verbosity hacking. Each problem therefore needs a different correction rather than a universal reward patch.
• Merge rather than maintain a larger fleet. The experts are combined with a two-stage SLERP procedure, producing a single model intended to cover the mixed corporate workload.
• Use different kinds of scoring. Deterministic verifiers score tasks with clear pass/fail conditions, while calibrated LLM judges handle cases that are harder to verify automatically. The team tracks an internal Arena, instruction-following quality, and function-calling quality.

What the reported results mean

In non-reasoning mode, the merged model scores 69.6 versus 65.8 for a baseline with roughly seven times more total parameters on the internal Arena. It also reports 0.85 versus 0.83 on instruction following and 0.79 versus 0.77 on function calling. The material says general dialogue benchmarks improve as well. In deployment, the model handles 50% of platform traffic, or 116 million requests per month.

The important claim is not that a smaller model universally beats a larger one. It is that a model optimized around an organization’s actual request distribution can be more useful per unit of serving capacity. Consolidation can simplify routing, monitoring, rollout, and capacity planning, while reducing the GPU fragmentation caused by keeping many specialized or legacy models alive. It also connects post-training directly to operating economics: better quality is valuable because it allows the unified model to absorb more traffic.

What remains unproven

The approach has a moving target. Internal traffic changes as applications add tools, prompts evolve, and new failure modes appear. A static benchmark can therefore overstate coverage if the retraining loop is slower than workload drift. The durability of the result depends on how quickly the evaluation set and post-training process respond to those changes.

The judge design also matters. A calibrated LLM judge can scale evaluation, but if it is too closely aligned with the optimization data, a model may learn to satisfy the judge rather than users. Long-term validation should therefore include human agreement on difficult cases, performance on newly arriving traffic, and coverage measured over time—not only a snapshot score.

Overall, the paper presents a practical model-maintenance pattern: mine production errors, separate conflicting objectives, merge the resulting experts, and evaluate against the real request mix. It reframes post-training as an ongoing serving and data-engineering loop rather than a one-off upgrade.

Source: Hugging Face Daily Papers