The memory ladder
Applying the table from the previous page to our own model narrows the choice to two arrangements.
| 8 GPUs | 16 GPUs | 32 GPUs | 64 GPUs | |
|---|---|---|---|---|
| data parallelism | 447.0 GiB | 447.0 GiB | 447.0 GiB | 447.0 GiB |
| ZeRO-1 | 153.7 GiB | 132.7 GiB | 122.2 GiB | 117.0 GiB |
| ZeRO-2 | 104.8 GiB | 80.3 GiB | 68.1 GiB | 62.0 GiB |
| ZeRO-3 | 55.9 GiB | 27.9 GiB | 14.0 GiB | 7.0 GiB |
A card holds 74.5 GiB. Data parallelism and ZeRO-1 never fit, at any number of GPUs. Both leave the weights and the gradients replicated on every card, which is four bytes per weight. Four bytes across 30 billion weights is 111.8 GiB, and that figure is the same on one card as on a thousand. ZeRO-2 fits from 32 GPUs upward. ZeRO-3 fits from 8.
That floor is set by the model size alone. Four bytes per weight fills a 74.5 GiB card exactly at 20 billion parameters, so a 20B model sits on the boundary and our 30B model sits past it. Dragging the model size in the widget below crosses that line.
The memory wall
The same four schemes on log axes, so the shape of each is visible across three orders of magnitude of GPU count. A solid line is the part that fits on one card; dashed is the part that does not. The faint horizontal lines are the replicated floors, and they are what stop two of the four from ever going solid.
Two arrangements survive for this model, and the choice between them is not about memory. ZeRO-2 needs 32 cards before it fits at all and holds communication at 2P. ZeRO-3 fits on 8 and leaves room to spare, but pays 3P for every step. On NVLink that difference is 0.13 seconds; over InfiniBand it is 1.2.
The pile we kept setting aside
Every number in this session has counted the sixteen bytes per weight, and every time activations came up they were deferred. They cannot be deferred forever, because on a real card the two piles share the same 74.5 GiB. Here they are together, with three generations of card drawn as ceilings rather than one.
The shape of the trade is the point. Sharding state is what ZeRO does and it works: the purple block falls from 447 GiB to under 60. But it does nothing to the amber block, and past a certain sequence length the amber block is the whole problem. That is where recomputation, sequence parallelism and the rest come in, and none of them are ZeRO.
A third answer: reversible architectures
Recomputation keeps a few checkpoints and recomputes forward from them. A reversible architecture goes further by changing the model so that each layer's input can be reconstructed from its output, which means the intermediate values need never be stored at all — only a boundary state at the end. The amber block nearly disappears rather than shrinking.
It is a different kind of decision from everything else in this session. ZeRO, offload and recomputation leave the model untouched and rearrange where numbers live. Reversibility constrains what the architecture may be, so it has to be chosen before training starts rather than tuned afterwards. Reported savings are large, but the comparison is only meaningful against a run doing the same thing, which is why published figures vary so much.