Overlapping communication with compute
The cost page measured communication as a share of step time. That share can be reduced without sending less data, by arranging for the sending to happen while the GPU is busy with something else.
The backward pass makes this possible. It works from the last layer of the model to the first, so the gradients for the last layer are finished long before the pass reaches the first layer. Those gradients can begin their journey immediately, while the earlier layers are still being computed.
Buckets
Gradients are collected into buckets, and a bucket is sent as soon as it fills, part way through the pass. The bucket size sets a balance.
| Bucket size | Effect |
|---|---|
| smaller | transfers begin earlier in the pass, giving more room to overlap |
| larger | the fixed cost of starting a transfer is paid less often |
Production settings are given in bytes, and a few hundred megabytes is a common choice. The widget below groups whole layers into each bucket so that twelve of them fit on one screen.
Two things come out of it. The tail is never smaller than a single bucket's transfer, because the last bucket only exists once the pass has finished and there is no computation left to hide it behind. And shrinking the bucket shrinks that tail, right up to the point where the per-transfer overhead starts to dominate.
There is a third effect that appears only on fast hardware. Smaller buckets keep helping while the link drains them as fast as the pass produces them. Past that point the transfers queue behind each other, the pass finishes first, and the remaining traffic is exposed no matter how it was divided.
Why this is the last lever, not the first
Overlap does not reduce traffic; it hides traffic behind work. If the traffic exceeds the work, no arrangement hides it, which is the ratio from the cost page. Overlap is what makes a 2P scheme cheap on a fast link and does nothing for a run that is already spending more time on the wire than in the arithmetic.