Collective operations
The averaging step on the previous page has a name, and there are three operations of its kind that appear throughout this session. Each one involves every GPU at once.
All-reduce combines a value from every GPU and gives the combined result back to all of them. Averaging gradients is an all-reduce.
Reduce-scatter combines the values in the same way and gives each GPU only one slice of the answer. Eight GPUs each end up holding one eighth of the averaged result.
All-gather is the reverse. Each GPU starts with one slice and ends with the complete set, assembled from everyone's slices.
| Operation | Each GPU starts with | Each GPU ends with |
|---|---|---|
| all-reduce | a full set of values | the combined full set |
| reduce-scatter | a full set of values | one slice of the combined set |
| all-gather | one slice | the full set |
How the exchange actually happens
The obvious way to combine a value held by every card is for every card to send it to every other card. Each one then has all four versions and can average them locally. It works, and it is worth drawing because it is the picture most people carry.
The trouble with the left picture is how it grows. With four cards each one sends three
copies. With eight it sends seven. The cost per card is
(N − 1) × P, so adding hardware makes the exchange more expensive
per card, not less.
| Cards | All-to-all | Ring | Ring, in GB |
|---|---|---|---|
| 4 | 3P | 2P | 120 |
| 8 | 7P | 2P | 120 |
| 32 | 31P | 2P | 120 |
| N | (N−1)P | 2P | 120 |
The ring column does not move. That is the whole reason rings are used: the cost per card is independent of how many cards there are. Thirty-two GPUs pay exactly what four pay.
It works because nothing is ever sent twice. In the first phase the data is cut into as
many chunks as there are cards, and each chunk makes its way around the ring accumulating
contributions, so that after N − 1 hops every chunk has been added up
somewhere — and each card is holding the finished version of exactly one chunk. That is
a reduce-scatter. The second phase passes those finished chunks round the ring again until
everyone has all of them, which is an all-gather.
Reading the two pictures together
The left diagram shows what the operation achieves: every card ends up informed by every other. The right shows how it is paid for. They are the same operation, and the ring is not an approximation of the all-to-all — the result is bit-for-bit what the all-to-all would have given.
The equivalence that makes the rest work
A reduce-scatter followed by an all-gather produces exactly what an all-reduce produces. This equivalence is the reason the next two sections work.
It also explains the cost, and it is the same two phases just described. Each GPU sends about one copy of the data during the first phase and receives about one copy during the second, giving a total of 2P.
The same averaging, taken apart
On the data parallelism page, phase 2 averaged four gradients and the result appeared in all four panels at once. That was an all-reduce, and it was shown as a single move because nothing before this page explained how a move like that is carried out.
Now take the identical numbers and do it the long way. Batch 1 produced these four gradients, one per card:
| Card | w0 | w1 | w2 | w3 |
|---|---|---|---|---|
| GPU 0 | +1 | −1 | −1 | −1 |
| GPU 1 | 0 | 0 | −2 | 0 |
| GPU 2 | −1 | +1 | −3 | +1 |
| GPU 3 | −2 | +2 | −4 | +2 |
| sum, then ÷ 4 | −0.50 | +0.50 | −2.50 | +0.50 |
The bottom row is what the previous page displayed. The six hops below are how a ring actually produces it.
The chunks here are the columns
The gradient has one entry per weight, so it splits into four pieces along its length: w0, w1, w2, w3. Four cards, four columns, and each card will end up responsible for one of them. A chunk is a slice of the gradient vector, not a separate quantity — every card starts holding all four slices, and what differs between cards is the values.
The ring runs 0 → 1 → 2 → 3 → 0, so GPU 0's left
neighbour is GPU 3. On hop 1 each card sends the column carrying its own index, and that
index shifts back by one on each hop after, which is what stops a column crossing the same
link twice.
Because the numbers are small and repeat, each card also shows which cards are
inside every value. A cell reading 0·3 has GPU 0 and GPU 3 in it
and is still missing two. A column is finished when it reads
0·1·2·3.
Is this mixing the weights together?
It can look that way, because GPU 0 sends something labelled w0 to GPU 1 and the value lands in GPU 1's w0. Three things keep that from being what it appears to be.
No weight ever moves. Every card holds 3 5 2 4 from step 0
to step 8 and does not touch them until step 9. Nothing crossing a wire is a weight. What
travels is gradients — opinions about how the weights should change.
w0 on one card and w0 on another are the same weight. All four cards hold identical copies of weight 0. What differs is each card's estimate of how weight 0 should move, because each card saw two different samples. Combining those four estimates is the entire purpose of the exchange.
A column never meets a different column. When a value arrives it is added into the receiver's copy of that same position. Column 0 only ever meets column 0.
Why the index is confusing
The label w0 is doing two unrelated jobs. It names which weight the number refers to, and it also names which card will end up owning that column after the scatter. Those two facts share an index by convention, not because they have anything to do with each other. GPU 3 ending up with the w0 total does not mean GPU 3 has acquired weight 0 — it means GPU 3 was assigned the job of finishing that one sum.
Every addition, written out
The objection worth answering head on: it looks like w0 is being added to w1. It is not, and the quickest way to settle it is to write down all twelve additions and check the column index on both sides of every plus sign.
Here is what each card believes at the start. Read a row as one card's opinion about each of the four dials.
| Card | w0 | w1 | w2 | w3 |
|---|---|---|---|---|
| GPU 0 | +1 | −1 | −1 | −1 |
| GPU 1 | 0 | 0 | −2 | 0 |
| GPU 2 | −1 | +1 | −3 | +1 |
| GPU 3 | −2 | +2 | −4 | +2 |
| column totals we want | −2 | +2 | −10 | +2 |
Hop 1 — four wires, at the same instant
Now each card on its own, which is the view that settles the question. Take GPU 1 during hop 1:
| Its column | Before | What happened | After |
|---|---|---|---|
| w0 | 0 | received w0 = +1 from GPU 0, adds 0 + 1 | +1 |
| w1 | 0 | sent a copy to GPU 2 — its own value is untouched | 0 |
| w2 | −2 | nothing | −2 |
| w3 | 0 | nothing | 0 |
The +1 came from w0 plus w0. GPU 1's w1 played no part in that addition
— it left on a different wire, headed to a different card. Sending is a copy; it does
not remove or alter anything.
The same view of GPU 3, where the numbers look least alike:
| Its column | Before | What happened | After |
|---|---|---|---|
| w0 | −2 | nothing | −2 |
| w1 | +2 | nothing | +2 |
| w2 | −4 | received w2 = −3 from GPU 2, adds −4 + (−3) | −7 |
| w3 | +2 | sent a copy to GPU 0 — own value untouched | +2 |
GPU 2's opinion of w2 was −3 and GPU 3's was −4. Two estimates of the same dial, different because GPU 2 saw samples 5 and 6 while GPU 3 saw samples 7 and 8. There is no reason two cards looking at different data should agree, and if they did agree there would be nothing to combine.
All twelve additions
Every addition performed during the entire reduce-scatter. Check the column index on each side of every plus sign.
GPU1.w0 = GPU1's w0 ( 0) + GPU0's w0 (+1) = +1 GPU2.w1 = GPU2's w1 (+1) + GPU1's w1 ( 0) = +1 GPU3.w2 = GPU3's w2 (-4) + GPU2's w2 (-3) = -7 GPU0.w3 = GPU0's w3 (-1) + GPU3's w3 (+2) = +1
GPU1.w3 = GPU1's w3 ( 0) + GPU0's w3 (+1) = +1 GPU2.w0 = GPU2's w0 (-1) + GPU1's w0 (+1) = 0 GPU3.w1 = GPU3's w1 (+2) + GPU2's w1 (+1) = +3 GPU0.w2 = GPU0's w2 (-1) + GPU3's w2 (-7) = -8
GPU1.w2 = GPU1's w2 (-2) + GPU0's w2 (-8) = -10 complete GPU2.w3 = GPU2's w3 (+1) + GPU1's w3 (+1) = +2 complete GPU3.w0 = GPU3's w0 (-2) + GPU2's w0 ( 0) = -2 complete GPU0.w1 = GPU0's w1 (-1) + GPU3's w1 (+3) = +2 complete
Twelve additions, and in all twelve the column index is identical on both sides. It is not a convention that happens to hold — a received column is always added into the same-numbered local column, so w0 meeting w1 cannot occur.
Where the confusion comes from
Listing the hop as four rows puts GPU 1 at the end of one row and the start of the next:
GPU 0 → GPU 1 w0 then GPU 1 → GPU 2 w1. That
reads like a chain — w0 goes in, w1 comes out. It is not a chain. GPU 1 is receiving
on one wire and sending on another at the same moment, and the second row would happen
identically even if the first did not exist.
Why a card does not forward what it just received
GPU 1 receives w0 during hop 1. It does not pass w0 on until hop 2. The reason is that the w0 total is not finished — it holds opinions from cards 0 and 1 and is still missing 2 and 3. Meanwhile GPU 1 has something that has begun no journey at all: its own w1. So that is what goes out this hop.
Each hop, every card forwards whichever column is due to move next, which is why the column sent and the column received are never the same one.
| Hop | Where it is | Value | Opinions inside |
|---|---|---|---|
| start | GPU 0 | +1 | 0 |
| 1 | arrives GPU 1, which adds its 0 | +1 | 0·1 |
| 2 | arrives GPU 2, which adds its −1 | 0 | 0·1·2 |
| 3 | arrives GPU 3, which adds its −2 | −2 | 0·1·2·3 |
| ÷ 4 | −0.50 | the averaged w0 |
And −2 is exactly the w0 column total from the first table on this
section: +1 + 0 + (−1) + (−2). The ring did not compute anything
different from the plain column sum. It only chose an order and a schedule for doing the
additions so that every wire carries one number per hop.
| Column | Route | Total | ÷ 4 | Ends on |
|---|---|---|---|---|
| w0 | GPU 0 → 1 → 2 → 3 | −2 | −0.50 | GPU 3 |
| w1 | GPU 1 → 2 → 3 → 0 | +2 | +0.50 | GPU 0 |
| w2 | GPU 2 → 3 → 0 → 1 | −10 | −2.50 | GPU 1 |
| w3 | GPU 3 → 0 → 1 → 2 | +2 | +0.50 | GPU 2 |
Four notes, each travelling the ring in the same direction, each one hop behind the last. That staggering is the whole design: it keeps all four wires busy at once, and it is why each card is always sending one column while receiving a different one.
Why do it this way at all?
The sums above could be computed by having every card send its whole gradient to every other card. Four cards, three copies sent each, and everyone can add up all four columns locally. The result is the same.
It costs more, and the cost grows with the cluster. Sending to everyone is
(N − 1) × P per card, so a run on 32 GPUs sends 31 copies. The ring
sends 2P regardless of N. The rotation exists to keep every link carrying exactly one number
per hop, so that no card is ever waiting on a queue behind three other transfers.
That is the whole justification. Not correctness — both approaches give identical numbers — but cost.
The whole chain, end to end
This runs the same batch as the previous page, but without skipping anything. Step 0 splits the batch, step 1 produces the four local gradients, steps 2 to 7 are the six ring hops, step 8 divides, and step 9 applies the update. The previous page compressed steps 2 through 8 into a single click; here they are opened up.
The thing to take from the table is that no card ever sends a whole set. Each hop moves one chunk per card, three hops in each phase, and that totals about one copy of the data sent per card per phase. Two phases, 2P.
Why this matters for what follows
Because an all-reduce already contains a reduce-scatter, stopping halfway is free. ZeRO-2 does exactly that: it runs the first phase, leaves each card holding its own slice, and never pays for the second. The saving is not a new technique — it is declining to finish an operation whose first half was all that was needed.
Three operations, then, and one identity between them. Every scheme on the pages that follow is built from these pieces, and the only question that ever changes is which slices a card keeps and which it has to ask for.