The cost of communication

Sending data between GPUs takes time, and that time is not free. While a GPU is waiting for numbers to arrive, it is not computing. The question for any distributed run is whether the waiting is small compared with the work.

Our model gives P of 60 GB, so data parallelism moves 120 GB per GPU on every step. Wiring inside a single machine carries that in about a quarter of a second. A network cable between machines carries the same 120 GB in about two and a half seconds.

Each of those four figures is one division: bytes to move, divided by bytes per second the link carries.

NVLink      450 GB/s     120 GB / 450 = 0.2667  ->  0.27 s
                         180 GB / 450 = 0.4000  ->  0.40 s

InfiniBand   50 GB/s     120 GB /  50 = 2.4000  ->  2.40 s
                         180 GB /  50 = 3.6000  ->  3.60 s

And the two byte figures come from P. One copy of the parameters in 16-bit is 30 × 109 × 2 = 60 GB, so 2P is 120 GB and 3P is 180 GB. Nothing else is in those numbers — no overhead, no contention, no latency per message. They are the floor, and a real run is slower.

The 9× gap between the two rows is the same 9× from the terminology page: 450 against 50. Crossing a node boundary costs nine times as much per byte, and the table is just that ratio applied to two payload sizes.

Whether those numbers matter depends entirely on how long the computation itself takes. A step processing one million tokens on 64 H100 cards takes about 7.1 seconds of compute at a realistic utilisation. The same step on 64 B200 cards takes about 3.1 seconds, because the cards are faster.

That last column is communication divided by compute, which is the standard test for whether a run is limited by its arithmetic or by its wiring. If none of the transfer is hidden, the step takes the sum of the two, so 9.51 seconds on H100 and 5.53 seconds on B200.

Faster GPUs raise the ratio of communication to compute. The volume stays at 120 GB while the compute it hides behind gets shorter. This is the most important consequence of buying newer hardware, and it is why the arrangement of the run matters more for a model of this size than it did for earlier and smaller ones.

The ratio is still below 1 in both rows, so a transfer that runs entirely during the computation is still fully hidden. Overlapping the exchange with the backward pass is how that is arranged, and the rise from 34 percent to 77 percent is what turns that arrangement from an optimisation into a requirement.

Where 2P comes from, and when it is a lie

The six hops on the previous page are where this number is born. Each hop, a card sends one chunk, and a chunk is one N^th of the gradient. Three hops for the reduce-scatter and three for the all-gather, so with four cards:

The exact figure is 2(N − 1)/N × P. It is 1.5P on four cards, 1.75P on eight, and 1.97P on sixty-four. It approaches 2P from below and never reaches it, so quoting 2P is a convenient overestimate that gets truer as the cluster grows.

From the walkthrough to the bill

The previous page ran four cards, four columns and six hops on a gradient of four numbers. That structure is the real one — it does not change at scale. What changes is how big a column is. Set the run and every stage of the toy example picks up its weight in bytes and seconds.

The strip makes the ZeRO-2 argument concrete before we get there. Half the blocks are copper and do arithmetic; half are cream and only copy. At 8 GPUs on InfiniBand the cream half is 52.5 GB and 1.05 seconds per card per step, spent moving numbers that were already finished.

Work it out

Set the cards and the link, and read both bars against the same pinned axis. Each lane carries a faint vertical line showing where its other option would land — the other card generation on the compute lane, the other interconnect on the network lane — so the effect of a change is visible without switching back and forth.