The challenge of fitting MoE models locally
Running a large mixture-of-experts (MoE) model on a consumer PC is not only a compute problem. Its weights may exceed available GPU memory and system RAM, forcing the inference stack to move data among VRAM, host RAM, and SSD. TIDEWAY addresses this constrained setting. Its premise is that these memory resources should be coordinated rather than treated as isolated caches—a separation that, the authors say, can lead to thrashing during inference.
Memory management that follows the inference phase
MoE models activate selected experts rather than using every expert for every token. The access pattern also changes between prefill, when the input is processed, and decode, when the model generates tokens one by one. A layout that works well for one phase may be a poor fit for the other. If placement and transfers fail to account for that difference, expert weights can compete for space and move through the hierarchy inefficiently.
TIDEWAY combines three ideas:
- A coordinated pool: It manages host RAM and GPU VRAM as a unified pool for weight placement and scheduling, rather than as unrelated caches. The abstract notes that weights may spill to SSD on constrained devices, but does not claim that SSD is managed as part of the same unified pool.
- Phase-aware layouts: Memory arrangements adapt to the distinct access patterns of prefill and decode.
- Paging-aware execution: Its execution engine optimizes CUDA Graph execution in the presence of expert paging, aiming to keep execution efficient while experts are brought in as needed.
Reported performance
On representative consumer hardware and frontier MoE models, the authors report average speedups over the fastest baseline of 1.97× for end-to-end latency and 2.40× for time-to-first-token (TTFT), alongside 1.61× higher decode throughput. These figures describe the evaluations reported in the paper’s abstract; they should not be read as guarantees for every device, model, prompt length, or workload. The abstract does not provide the hardware configurations, model list, test conditions, or distribution of results needed to assess how performance varies across those settings.
Why it matters—and what remains open
TIDEWAY frames local MoE inference as a problem of managing data across a memory hierarchy, not simply maximizing GPU compute. Its focus on the different needs of prefill and decode also argues against assuming that one static memory arrangement is optimal throughout inference. If the approach generalizes across consumer systems, it could reduce waiting time and improve generation rates for people running large models locally, including applications that prioritize privacy or seek to limit cloud API use.
The practical gains will still depend on available memory, storage performance, model choice, and workload. The abstract alone does not establish deployment requirements or performance across device classes. The project’s code is public, making further examination of its implementation and evaluation conditions possible.
Source: Hugging Face Daily Papers
Comments
Checking sign-in status...
Loading comments...