Data parallelism

The simplest way to use eight GPUs is to give each of them a complete copy of the model and a different portion of the data. Each GPU reads its own examples, runs them through its own copy, and produces its own gradients. Those gradients differ from each other, because each GPU saw different text.

The copies then have to be brought back into agreement. Every GPU sends its gradients to every other GPU, all of them compute the average, and each applies that same average to its own copy. Because they started identical and applied an identical update, they remain identical. This arrangement is called data parallelism.

Watch it happen

Four GPUs hold one copy each of the same four weights, and a batch of eight samples is split two per GPU. The loss on a sample is L = 0.5 · ‖w − x‖², so that sample's gradient is w − x, and a GPU's gradient is the mean over its two samples. The update is w := w − 0.1g.

The batch of eight is split, not copied: samples 1 and 2 go to GPU 0, samples 3 and 4 to GPU 1, and so on. Each sample is seen by exactly one card, which is why the eight gradients that exist across the cluster are four means of two rather than eight copies of anything. The strip at the top of the widget shows the split for the current batch.

Each batch runs in three phases and the button walks one phase at a time. The working underneath the panels shows the actual arithmetic at each phase — which numbers are being subtracted from which, what is being averaged with what, and where the result lands.

So the answer to where the average goes is: back to everyone. All-reduce is not a gather-to-one-card operation. Every card ends the exchange holding the same averaged vector, and then every card does its own subtraction locally. No card is in charge, and no card has anything the others lack.

That is what keeps the copies identical. They began identical, and they applied the same number to the same starting value, so they end identical — not approximately, exactly. The drift figure stays at 0.000 however many batches you run.

Turn averaging off and phase 2 exchanges nothing. Each card applies its own gradient, the bottom rows stay different, and the copies separate on the first step. They are then four different models that happen to share a starting point.

Why the averaging is what makes it correct

The averaging step is not bookkeeping — it is the thing that makes the arrangement mathematically defensible. Averaging the gradients from eight GPUs, each of which processed 32 sequences, produces exactly the gradient that a single GPU would have produced from all 256 sequences at once.

The distributed run is therefore identical to a single-GPU run on a batch eight times larger. This is the property that lets a training recipe move from one GPU to many without changing what the model learns. Nothing about the result depends on how many cards were involved.

The batch the optimizer sees

It is the product of three numbers, and it is worth being able to state which one you are turning.

Accumulation steps are repeats of the forward and backward pass before the weights are updated, used when the desired batch is larger than what fits in memory at once. Gradients from each repeat are added up, and only then is the update applied.

So the three dials trade against different resources. Sequences per GPU costs activation memory, GPUs cost hardware, accumulation costs wall-clock time. Any of them can be turned to reach a target global batch, and the optimizer cannot tell which one you used.

What data parallelism does not touch is the memory from the first page. Every one of those eight cards is still carrying the full sixteen bytes per weight, and seven of the eight copies are redundant. That is the next page.