The arithmetic behind Cerebras’s 59-wafer plan for Kimi K3

What does it take to run a model with roughly three trillion parameters if an AI chip has only 44 gigabytes of on-chip memory? Cerebras published a detailed answer on 22 October, using the open-weight Kimi K3 model as an illustration. Its answer is 59 wafer-scale engines connected as a pipeline—not because 59 is a universal price of three trillion parameters, but because the structure of this particular model creates two different kinds of memory requirement.[1]

The example is unusually useful because it exposes the arithmetic behind a hardware supplier’s headline. It does not, however, document a measured production deployment of Kimi K3 on exactly 59 installed wafers. Cerebras explicitly presents the configuration as an illustrative mapping, with chosen assumptions about weight precision, request length and attention cache.[1]

Start with the weights

Kimi K3 has about three trillion parameters. The demonstration assumes four-bit weights, equivalent to half a byte per parameter before additional packing or metadata. That produces 1.5 trillion bytes of raw weights, or approximately 1.5 terabytes using decimal units. Divide across the model’s 93 layers and the arithmetic gives roughly 16.1 gigabytes per layer. Cerebras rounds to approximately 16 gigabytes.[1]

The layers are not operationally identical. Of the 93, Cerebras identifies 69 as linear-attention layers and 24 as dense-attention layers. A linear-attention layer can keep a comparatively compact state as a sequence grows. A dense-attention layer must keep the keys and values needed to attend to earlier tokens; the capacity requirement grows with active contexts and users. Both kinds may carry similar quantities of weights, but their working memory behaves differently.[1]

A 44-gigabyte wafer can therefore hold about two of the linear-attention layers at the assumed precision. Cerebras allocates 35 wafers for those 69 layers, rounding up because the final wafer is not completely filled. It gives each of the 24 dense layers its own wafer so that spare memory can hold the key-value cache. The accounting is 35 plus 24: a 59-wafer model pipeline.[1]

An assembly line for requests

It is tempting to imagine a token visiting one giant processor, waiting, and then moving to the next. The design instead resembles an assembly line. Each wafer retains its assigned weights. Intermediate activations pass between stages, and different customer requests can occupy different stages simultaneously. Cerebras cites approximately two microseconds for inter-wafer data movement by RDMA over Ethernet in its account.[1]

Each dense-layer wafer must hold a separate key-value cache for every active request. Cache capacity therefore limits concurrency, while micro-batch size trades individual response latency against total throughput. Keeping weights in local memory solves only one part of serving a large model.[1]

The hidden cost of a long conversation

The illustrative workload assumes an average request of 256,000 tokens with eight-bit key-value cache. That is a very large context, and it is significant because each active dense-attention request consumes memory beyond the weights. If requests are shorter, a wafer can potentially hold more concurrent cache entries. If the attention architecture changes, the entire wafer accounting may change. If precision changes, the simple 1.5-terabyte weight estimate no longer holds.[1]

The calculation makes a useful point about model size: three trillion parameters do not specify a hardware bill. The split between attention types, weight precision and the context carried for each user determines how much memory the service actually needs.[1]

Report an error or suggest a correction