Side by side
Everything the session established, in one table. Memory is per GPU at 30 billion parameters on eight cards; communication is per GPU per step.
| Bytes/weight | Per GPU | Traffic | Fits a 74.5 GiB card | |
|---|---|---|---|---|
| data parallel | 16.00 | 447.0 GiB | 2P | never, at any GPU count |
| ZeRO-1 | 5.50 | 153.7 GiB | 2P | never — 4-byte floor |
| ZeRO-2 | 3.75 | 104.8 GiB | 2P | from 32 GPUs |
| ZeRO-3 | 2.00 | 55.9 GiB | 3P | from 8 GPUs |
| Lever | Buys | Costs |
|---|---|---|
| more GPUs | divides the sharded bands | nothing in memory; the replicated floor does not move |
| offload to system memory | the optimizer state leaves the card | PCIe at 60 GB/s, a third of NVLink |
| overlap with the backward pass | hides most of the traffic | nothing, except the last bucket |
| recomputation | most of the activation memory | about a third more compute |
| 8-bit matmul | 12% of stored state, half the traffic | Blackwell hardware, and care around softmax |
The order to reach for them
The session suggests a sequence rather than a menu, because the cheap moves are also the early ones.
What this session did not settle
Four questions remain open, and each one needs a measurement rather than an argument.
| Question | What would settle it |
|---|---|
| ZeRO-2 on 32 GPUs, or ZeRO-3 on 8? | A measured step time for both on our real architecture, with activation memory included. |
| How many GPUs per node, and how many nodes? | The cost page shows a nine-fold difference between the two interconnects, so the answer follows from how much traffic can be kept inside a node. |
| Is 8-bit arithmetic committed from the start? | A short run in bf16 and in MXFP8 on the same architecture, comparing loss and step time. This commits us to Blackwell hardware. |
| Does any state go to system memory? | Whether the run is memory-bound or communication-bound once the stage is chosen. |
Why these four and not others
Each is a question the arithmetic cannot close. The memory figures are exact and the communication figures are floors, but step time depends on the real architecture, the real batch, and how well the overlap actually works — and the first question turns entirely on activation memory, which this session deliberately set aside. The last three are decisions about hardware and are therefore expensive to reverse, which is a reason to measure early rather than to argue longer.
What to take away
Three things, if nothing else survives.
The optimizer state is three quarters of the memory. Twelve of the sixteen bytes exist only for the update, are touched once per step, and were replicated on every card for no reason. Everything ZeRO does follows from noticing that.
Sharded state divides; replicated state does not. Whatever a scheme leaves replicated sets a floor that no amount of hardware will lower, which is why data parallel and ZeRO-1 can never fit this model however large the cluster gets.
Nothing here changes what the model learns. Every arrangement in the session computes the same gradient and applies the same update. They differ in where bytes live and how many cross a wire, which makes all of it an engineering decision and none of it a modelling one.