How Colibrì Uses SSDs to Run Giant MoE Models on Consumer PCs

Introduction

For extremely large language models, the first deployment problem is often not raw compute but storage capacity. A model may contain hundreds of billions or even trillions of parameters, while a consumer laptop has only a few dozen gigabytes of RAM. Colibrì, an open-source inference project, proposes a practical workaround: do not keep the entire model in fast memory. Store rarely used weights on an NVMe SSD and load them only when the router selects them.

The project is written in pure C and does not depend on a separate inference engine. According to the supplied material, it supports several model families and was initially designed around experiments with the 744B-parameter GLM-5.2. Its repository has attracted tens of thousands of GitHub stars.

The key idea: load experts on demand

Mixture-of-Experts models provide the opening for this design. Their total parameter count can be enormous, but each token activates only a subset of experts. In the example described by the project, GLM-5.2 has 744B total parameters and roughly 40B active parameters per token.

Colibrì divides the model into storage tiers:

• Dense components such as attention, embeddings and shared experts remain in RAM. In the supplied GLM-5.2 configuration, quantized resident weights take about 9.9GB;
• The large pool of routed experts is stored primarily on SSD;
• Experts selected by the router are fetched into RAM, and can be promoted to VRAM when a GPU is available.

This resembles just-in-time loading for model weights. Placement affects latency, not model behavior: the router still makes the same selection, and the framework does not intentionally remove experts simply because the machine has less memory.

Caching is as important as capacity

Reading from an SSD for every token would make generation painfully slow. Colibrì therefore uses an LRU cache to retain recently used experts and tracks access frequency so that frequently selected experts receive higher cache priority. It also attempts to predict experts for the next layer. The project cites meaningful correlation between neighboring routing decisions and uses that signal to prefetch data while the current layer is still computing.

The framework can also place a second copy of the model on another SSD, allowing reads to be distributed across drives. Project-reported tests put cold-cache performance on a machine with 25GB of RAM at roughly 0.05–0.1 tokens per second. A CPU-only system with 128GB of RAM reportedly reaches about 1.8 tokens per second, while a setup with six RTX 5090 GPUs reaches approximately 5.8–6.8 tokens per second when more experts can remain in fast memory. These are project figures under particular conditions, not universal hardware benchmarks.

Why it matters—and what it does not solve

Colibrì’s main contribution is not making an SSD as fast as GPU memory. It changes the deployment equation: SSDs provide capacity, RAM acts as a larger cache, and VRAM handles the hottest data when available. The project reportedly supports models ranging from smaller Qwen and OLMoE variants to GLM, DeepSeek, Inkling and the 2.8T-parameter Kimi K3, although the largest models still require hundreds of gigabytes or even terabytes of storage.

This makes previously inaccessible models easier to experiment with on consumer hardware. It does not turn a laptop into a high-throughput inference server. Startup time, cold-cache latency and sustained generation remain tied to SSD bandwidth, RAM size, expert reuse and model architecture. For researchers, Colibrì is an interesting systems experiment; for everyday users, it is better understood as a low-cost way to explore giant models than as a production solution.

Source: QbitAI