HybridInfer Routes LLM Requests by Phone Thermal Headroom

Introduction

Running a language model locally offers clear benefits: user data can remain on the phone, the system can work offline, and each query avoids a cloud inference fee. Yet HybridInfer suggests that the main mobile constraint is not always a gradual loss of speed. On a flagship Snapdragon device, sustained on-device generation destabilized the GPU inference runtime, sometimes causing crashes or silently leaving it unusable.

That finding changes the design problem for model routers. A system moving requests among device, edge, and cloud tiers must consider not only quality, latency, and cost, but also the phone’s remaining thermal headroom.

How HybridInfer works

The proposed hierarchy contains three models: Llama 3.2 3B on the device, retrieval-augmented Llama 3.1 8B at the edge, and GPT-4o in the cloud. HybridInfer represents each request using the phone’s thermal headroom and an estimate of query complexity. An offline-trained Q-learning policy then chooses the tier for the next inference request.

The reward balances answer quality, latency, monetary cost, and thermal risk. It also includes a locality bonus for on-device execution. This detail is central rather than cosmetic: without the bonus, the learned policy preferred to offload every request. Such a strategy may reduce local hardware pressure, but it removes the privacy, offline availability, and cost advantages that motivate on-device inference in the first place.

Results on real hardware

The evaluation used an Android harness on a Samsung Galaxy S25+ with a workload of 210 prompts and frozen reference answers. Compared with two hand-tuned heuristics, the learned router achieved significantly higher quality, with a paired Wilcoxon test reporting p < 0.02. It also had the lowest cost among the adaptive conditions.

The study does not claim that the local model is universally better. For queries that the device can serve, an always-on-device setup reached comparable per-query quality. However, it was three to six times slower and failed on long queries. HybridInfer’s advantage therefore comes from latency, reliability, and coverage—not from improving the intrinsic quality of the smallest model.

Why it matters

HybridInfer treats thermal behavior as part of inference policy rather than as a separate hardware-monitoring concern. For mobile AI products, this suggests that routing decisions should combine model capability with device state, runtime stability, privacy preferences, and cost constraints.

The evidence remains bounded by one phone, 210 prompts, and one three-tier model configuration. Results may differ across chipsets, thermal policies, runtimes, and workloads. Still, the release of the Android measurement harness and the full routing pipeline makes broader replication possible.

The broader lesson is practical: reliable on-device AI may not come from keeping every request local. It may come from knowing when local execution is appropriate—and when a temporary handoff to the edge or cloud protects the user experience.

arXiv