When the model is bigger than the machine
A single graphics card has a fixed amount of memory. The largest cards available to us hold 80 GB. The model has 30 billion numbers in it, and every one of those numbers has to be stored somewhere while the model is being trained. This session is about what happens when the thing you are training is larger than the machine you are training it on.
The answer turns out to be a sequence of increasingly clever ways to split the storage across several cards. Data parallel splits the work but not the storage. ZeRO splits the storage too, one category at a time, and each stage buys memory by spending network traffic. We work the whole arc by hand on a model small enough to hold in your head.
What is actually stored for one weight during training, and why it is sixteen bytes rather than two.
Four multiplications, four cycles on one CPU thread, one cycle on a GPU. Why that works, and what it assumes.
Why an unfused chain of small operations pays six trips to memory for three operations' worth of arithmetic.
Seven words with numbers attached: GPU, node, world size, rank, interconnect, collective, and P.
Batch size grows activations. GPU count grows replicas. Getting these two apart before the arc begins.
Copy the model onto every card, split the batch, average the gradients, and see why the result matches a single-GPU run.
All-reduce, reduce-scatter, all-gather — and the identity between them that every later saving depends on.
What the six hops actually cost in seconds, against what the step actually computes.
Eight identical copies of 447 GiB, and the three stages that take them apart one class of state at a time.
Which arrangements fit a card, at which GPU counts, and the replicated floor that no amount of hardware lowers.
Moving the optimizer state off the card entirely, and the PCIe bill that comes with it.
The two libraries that implement ZeRO, what each covers, and why the choice is practical rather than mathematical.
Sending each bucket of gradients while later layers are still computing, and the one bucket that can never be hidden.
MXFP8, the 12 percent it takes off stored state, and the half it takes off the wire.
Every scheme and lever in one table, the order to reach for them, and the four questions only a measurement can settle.
Build plan for the notebook: what to measure, what the numbers should come out as, and what the README has to argue.
A flat parameter vector, three collectives charged at ring cost, and a gradient check against finite differences.
Three flags describe all of them. Where the third P comes from, and the mistake that hides it.
Identical learning, and measured bytes against the formulas — both as tests that pass.
The measured sweep, the four figures to produce, and the three things worth admitting in the README.