llama-modchip Tries to Run a 284B MoE Model in 64GB of RAM

Introduction

Large-model inference is usually constrained by GPU memory and memory bandwidth. Once a model reaches tens or hundreds of billions of parameters, ordinary desktops are largely excluded from experimentation. llama-modchip takes a different route: it moves the focus to system RAM and CPUs, then combines aggressive quantization, cache-aware kernels and runtime tools for sparse mixture-of-experts (MoE) models. According to the repository, a 284.33B-parameter DeepSeek-V4-Flash variant can run on a 64GB system with no GPU.

The figures in this article come from the project documentation. The supplied material does not include independent third-party replication, so the throughput, memory and quality numbers should be treated as project claims rather than established benchmarks.

Key points

• Very aggressive compression. The project provides a Fractal-64 build of DeepSeek-V4-Flash at about 56.56 GiB and an effective 1.71 bpw. It describes the model as having 284.33B total parameters and 13.17B active parameters, and says the build fits within roughly 62.5 GiB of usable memory on a 64GB Linux system.
• CPU-oriented kernels. Its ggml/llama.cpp patches target AVX-512, AVX2 and selected ARM platforms. The design uses multi-level register and cache tiling, together with streaming techniques intended to stop cold expert weights from evicting frequently used KV-cache data.
• MoE sparsity does not eliminate all computation. The default configuration evaluates six routed experts and one shared expert per layer. A runtime Top-K setting can reduce the number of routed experts to improve throughput, but the project presents this as a quality-versus-speed trade-off.
• Smaller checkpoints are also supported. For Nemotron-3.5-30B-A3B, the repository lists a 14.35 GiB, 3.75-bpw build and reports 25.0 to 60.4 tok/s on a 16-core Zen 5 system. It also describes CPU-plus-GPU and multi-channel workstation configurations.
• It is not a drop-in model format. Fractal-64 relies on a custom GGML type, identified as type 40, plus matching kernels. Standard binaries of llama.cpp, Ollama or LM Studio will not recognize these tensors. The project supplies scripts, prebuilt binaries, a web interface, benchmarks and an integration path for Claude Code.

Why it matters—and what remains unclear

If the reported results are reproducible, the project suggests that the practical boundary for large MoE models may depend less exclusively on GPU VRAM. Expert sparsity, very low-bit weights and CPU memory bandwidth can create another, slower but accessible operating point. This could be useful for local development, model experimentation and agent workflows on machines without high-end GPUs. Its KV snapshots and context-restore tooling also show an interest in repeated tool-use sessions rather than only single-turn chat.

There are important caveats. A 1.71-bpw representation raises questions about accuracy, robustness and task-specific degradation. The repository mentions quality and tool-calling evaluations, but the supplied material does not provide complete baselines, prompts, hardware methodology or independent results. Anyone testing it should measure peak resident memory, long-context behavior, the exact definition of throughput and performance on coding and tool-use tasks. For now, llama-modchip is best understood as an ambitious CPU-inference engineering experiment, not a general replacement for GPU serving.

Source: Hacker News