What multiplies, and what doesn't
Before splitting anything across cards, be exact about which piles of memory grow when you turn which dial. Two dials, two different answers, and mixing them up makes the rest of this session unreadable.
Batch size does not copy the model
A batch of 8 means 8 samples flow through the same weights. The weights are read, not consumed. Sample three does not need a private copy of the matrix, any more than eight people reading one signpost need eight signposts.
What does scale with batch size is activations — the intermediate values each sample produces on its way through the network, kept because the backward pass needs them. A batch of 8 holds 8 sets. A batch of 64 holds 64. That is real memory, and it is why raising the batch size is what usually makes a card run out. But it is a separate pile from the sixteen bytes per weight.
| Pile | Scales with | Batch 8 → 64 |
|---|---|---|
| params, gradients, optimizer states | parameter count | unchanged |
| activations | batch size × sequence length | 8× |
Which pile this session is about
Data parallel and ZeRO both work on the first row. Activation memory is a real problem with its own solutions — gradient checkpointing, sequence parallelism — and it is not what we are doing here. Every number from this point counts the sixteen bytes per weight.
GPU count does copy the model
The other dial behaves completely differently. Put the model on 8 cards so they can each work on part of the batch, and each card needs the weights locally to do its forward pass. Eight cards, eight replicas.
Eight is the number that matters, and it is the number the next few pages attack. Eight identical sets of optimizer states sitting on eight cards, when between them they only ever needed one set spread out. Nothing about the arithmetic requires that duplication. It is an artefact of how data parallel was built, and removing it is what ZeRO does.