Edge0 Uses Trained Prerouting to Serve 35B MoEs from SSD

Introduction

Running a large mixture-of-experts model on consumer hardware is often presented as a compute problem, but storage and memory can be the harder constraint. An MoE activates only a small subset of experts for each token, which lowers the arithmetic performed per step. It does not, however, remove the need to store the model’s full expert pool. The supplied example puts a 35B-class model at about 19.5GB in 4-bit form, already close to the capacity of many consumer systems.

Edge0 addresses this bottleneck by placing expert weights on an SSD and trying to overlap storage reads with model computation. According to the material, the system serves a 35B MoE at 20 token/s on a single 24GB machine, with peak active memory below 3GiB. Across five public benchmarks, it is reported to remain within a few points of its FP16 teacher on average.

What the system changes

• It targets the timing problem in offloading. The experts needed by layer N+1 normally cannot be known until layer N has produced its output. Waiting for that routing decision before starting SSD reads leaves too little time to hide I/O behind computation.
• It adds a prerouter. A lightweight head attached to each layer predicts the next layer’s expert selection one token in advance, allowing the engine to begin staging weights earlier.
• It treats the prediction as the route. This is an important distinction. The system does not predict a route, compare it with the original router, and then recover from mistakes. It executes the predicted experts directly. Thus, the project’s statement that the route is “100% accurate” describes internal consistency between prediction and execution, not 100% agreement with the original routing policy.
• It uses a recovery LoRA. Quantization and replacing the original routing behavior can both reduce quality. Edge0 trains an unmerged LoRA on the student execution path so that the adapter learns to compensate for the deployed, quantized and prerouted model rather than the original teacher path.

Why it matters—and what remains unclear

If the reported measurements can be reproduced, Edge0 points to a broader design direction for edge inference. Instead of relying only on smaller models or more aggressive compression, it treats storage bandwidth, expert scheduling, routing and adaptation as one system problem. Devices with limited RAM but reasonably fast NVMe storage could potentially run models that would otherwise not fit in memory.

The central trade-off is also the most important caveat. The supplied material does not provide per-layer prediction statistics, the distribution of routing errors, the cost of fetching the wrong experts, or latency distributions across SSDs and sequence lengths. Because the predicted route is executed directly, an incorrect prediction is not automatically corrected by falling back to the original router. It may instead appear as a quality loss while throughput remains unchanged. Average benchmark scores and average token rates are therefore not enough: first-token latency, per-token jitter, long-context behavior, SSD bandwidth pressure and layer-by-layer routing differences would all matter for deployment.

Edge0 is consequently interesting less as a claim that SSDs make MoE inference free, and more as an attempt to coordinate trained routing prediction, streaming weight access, quantization recovery and low-memory execution. The framework, checkpoints and adapters are described as open source, which should make independent reproduction possible. Still, the broader conclusions should be based on the full paper, implementation and detailed systems measurements rather than the headline figures alone.

Source: Hugging Face Daily Papers