03 / GROUNDWORK

Terminology

Seven words are used throughout the session, and each is given here with a number attached.

The seven, at a glance
Word	Is	Number
GPU	one graphics card	80 GB
node	one machine holding several GPUs	8 GPUs
world size	total GPUs in the run	32 on four nodes
process rank	index of one GPU in that set	0 to 31
interconnect	the wiring between GPUs	450 or 50 GB/s
collective	an operation every GPU runs together	—
P	one copy of the parameters, in bytes	60 GB

GPU

One graphics card, and the unit of compute we buy. In this session every GPU has 80 GB of memory, which is 74.5 GiB.

Node

One physical machine holding several GPUs, almost always eight.

To make that concrete: a DGX B200 is built with eight Blackwell GPUs and carries 1.4 TB of GPU memory between them, with 64 TB/s of HBM3e bandwidth and 14.4 TB/s of all-to-all bandwidth across the eight. It is rated at 72 petaFLOPS for training and 144 for inference.

8Blackwell GPUs

1.4 TBGPU memory in the box

14.4 TB/sall-to-all between them

72 PFtraining performance

Figures from the NVIDIA DGX B200 datasheet.

That 1.4 TB is the number to hold on to. The 480 GB of training state from the first page does not fit on one card, but it fits inside one of these boxes about three times over — and everything inside the box talks over the fast wiring rather than the slow one.

Six cards' worth of memory, and a node holds eight. This is why the first thing to try is splitting across a single node.

A note on generations

The 80 GB card and the 450 GB/s NVLink figure used throughout this session describe the previous generation, and every number on the following pages is computed against them. A B200 card holds 180 GB and its links are far faster. The ratios are what the session is teaching, and those hold across generations even as the absolute figures improve.

World size

The total number of GPUs taking part in a training run. A run on four nodes has a world size of 32.

Process rank

The index of one GPU within that set, numbered from 0. In a world size of 32 the ranks run from 0 to 31. This word is used elsewhere in mathematics for the number of independent rows in a matrix, and that meaning is unrelated to this one.

Ranks are numbered straight through, ignoring node boundaries. Rank 7 and rank 8 are neighbours by number but sit on different machines, and the wire between them is the slow one.

Interconnect

The wiring that carries data between GPUs. Inside a node the wiring is called NVLink and carries roughly 450 GB per second. Between nodes it is a network cable, usually InfiniBand, and carries roughly 50 GB per second.

450GB/s inside a node

50GB/s between nodes

9×the penalty for leaving the box

Collective

An operation that every GPU in the run performs together, at the same time, on data that each of them holds a piece of. All-reduce, reduce-scatter and all-gather are the three that matter here, and each gets walked through by hand when it first appears.

P

The size of one complete copy of the model's parameters, measured in bytes. For a 30 billion parameter model held in 16-bit format, P is 60 GB. Communication in this session is always measured in multiples of P, so a cost of 2P means 120 GB crosses the wire for each GPU on every step.

Why measure in P rather than gigabytes

Writing a cost as 2P instead of 120 GB keeps the comparison honest across schemes. Every method on the following pages moves some multiple of one parameter copy, and stating it that way means the ranking does not change when the model size does.

P, in the units used later
Cost	Bytes per GPU per step	At 450 GB/s
P	60 GB	0.13 s
2P	120 GB	0.27 s
3P	180 GB	0.40 s

Those times are the floor, not the estimate — they assume the link runs at its rated speed with nothing else competing for it. They are here so that the multiples of P on later pages land as seconds rather than as abstractions.