FreeToken Reframes Personal Computers as Elastic MoE Serving Platforms

Introduction

Open-weight models are becoming increasingly capable, but serving them still tends to assume datacenter infrastructure. On a personal machine, the limiting factor may be GPU memory, system-memory bandwidth, CPU-GPU transfers, or the growing amount of state maintained by agentic applications. FreeToken approaches the problem from a different angle. Rather than treating a personal computer as a smaller GPU, it presents the entire machine as an elastic inference platform with several resources that can be coordinated at runtime.

Key points

• A serving stack designed around MoE behavior. FreeToken brings model layout and loading, expert residency, CPU-GPU execution, agent-state reuse, and runtime memory management into one design. The goal is not to optimize a single kernel while leaving the rest of the serving path unchanged.
• Bandwidth-adaptive placement. Personal computers have very different balances of GPU memory, system memory, processor capacity, and interconnect bandwidth. FreeToken avoids committing to one permanent offloading plan and continuously maps computation and model state onto the resources that are actually available.
• Support for changing agent workloads. Coding agents and tool-using agents can change their execution pattern over time. Their expert-access behavior and state requirements may therefore shift during a session. FreeToken treats this variability as a runtime concern and includes mechanisms for reusing agentic state.
• A broader local serving range. The project supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. The paper cites serving a 35B model on a laptop, a 284B model on a gaming desktop, and the 753B GLM-5.2 on one workstation GPU.

Why it matters

FreeToken is not merely about fitting a larger parameter count into a smaller device. Its central idea is that local inference is also a bandwidth and state-management problem. When the full model cannot remain in GPU memory, a serving system must decide which experts should stay resident, which states can be reused, and when CPU, GPU, or other memory resources should participate in execution. For MoE models, only a subset of experts is activated for each token, but moving expert weights can still dominate the user experience. Dynamic placement can therefore be more useful than a simple, fixed partition.

This approach could help move open weights from something users can download to software they can deploy locally. The supplied material does not include controlled throughput, time-to-first-token, generation-speed, or energy comparisons across the listed devices. The reported model sizes should consequently be read as demonstrations of serving scope rather than universal performance guarantees. FreeToken is released through its project site and code repository, while practical results will depend on the model, hardware configuration, and workload.

Source: Hugging Face Daily Papers