How Cerebras Maps a 3-Trillion-Parameter Model Across 59 Wafers

For trillion-parameter models, the first deployment problem is often memory rather than raw compute: where can the weights and request state live? In a technical post, Cerebras uses the roughly 3-trillion-parameter open model Kimi K3 to illustrate how a model can be distributed across multiple WSE-3T wafer-scale processors. The performance and benchmark comparisons mentioned in the original post are Cerebras’ own description of the example, not independent verification.

Splitting the model by layer

A single WSE-3T provides 44 GB of on-chip memory, while the weights of a trillion-parameter model can require terabytes. Cerebras therefore assumes 4-bit weight quantization and estimates storage by model layer. For the Kimi K3 example, dividing 3 trillion parameters across 93 layers produces an average layer footprint of about 16 GB.

The layers cannot all be placed in the same way. Kimi K3 is described as having 69 linear-attention layers and 24 dense-attention layers. Linear attention keeps a relatively small, fixed-size state for each request, so two such layers can fit on a 44 GB wafer in the example. Dense attention must retain a KV cache that grows with the request context, so the design places one dense layer on a wafer and reserves the remaining memory for that cache. This leads to 35 wafers for the linear layers and 24 for the dense layers, or 59 wafers in total.

Moving activations, not weights

Once the model is mapped, its weights stay on the wafers assigned to each stage. A request travels through the model like an item on an assembly line: one stage computes an activation, sends it to the next stage, and continues with subsequent tokens. Cerebras says the stages communicate through RDMA over Ethernet, with an indicated latency of roughly two microseconds per hop.

The important architectural point is that complete layers can usually fit directly on a wafer. As a result, weights do not need to be repeatedly moved between chips. The inter-wafer traffic is primarily the activation data produced by one stage and consumed by the next.

Keeping the pipeline full

A pipeline does not imply that only one request is being processed at a time. While one request is in a later stage, another can be in a middle stage and a third can enter an earlier stage. In production, Cerebras says it uses much finer-grained stages than the simplified layer-by-layer illustration. Each stage can also work on a micro-batch rather than an individual request.

Smaller micro-batches can favor per-request latency, while larger ones can improve aggregate throughput. The amount of memory left for KV cache on dense-layer wafers sets another limit: every in-flight request needs its own cache state. Thus, the deployment problem involves balancing weight storage, cache capacity, concurrency, communication, and latency.

Why the design matters

The example shows one way wafer-scale systems can extend beyond the capacity of a single accelerator. Adding wafers increases available memory and compute, while pipeline execution helps overlap work across stages. For system comparisons, however, batch size or peak tokens per second alone is not enough. Throughput per unit of power, measured at a comparable generation speed, may be a more useful operational metric.

The 59-wafer figure should also be read as a configuration example, not a universal requirement for every 3-trillion-parameter model. The result depends on the assumed context length, quantization, layer structure, cache format, and request mix.

Source: Hacker News