10 / THE ARC

Offload to CPU and NVMe

A GPU is not the only memory in the machine. A node has system memory, usually far more of it than the GPUs have, and it has solid-state storage beyond that. State that is needed rarely can be kept in those slower places and brought in when required.

one node, three places to keep a number GPU memory 80 GB per card 3,350 GB/s to the lanes system memory 2 TB, up to 4 reached over PCIe NVMe tens of TB slower again PCIe 60 GB/s for comparison, the wires from the terminology page NVLink 450 PCIe 60 InfiniBand 50 PCIe sits next to the network cable, not next to NVLink
Capacity rises to the right and speed falls. Offload is the decision to move something rightward.

The optimizer state is the natural candidate, because it is twelve of the sixteen bytes and it is touched exactly once per step. Holding it in system memory removes it from the GPU entirely.

Two versions of the same idea

The simpler one parks the state in system memory and copies it to the GPU for the update. The more effective one performs the update on the CPU as well, so the state never moves and the GPU is freed of that work altogether.

park it, copy it in for the update system memory 360 GB of state copy in copy back GPU does the update both directions over PCIe update where it already lives system memory does the update gradients in GPU forward and backward only one direction, and only the gradients the state never crosses PCIe in the second version — only the numbers that were going to move anyway
The second version is not a refinement of the first. It moves a different, much smaller thing.
12bytes per weight, offloaded
×
30 Bweights
=
360 GBoff the card, into system memory

Copying that back and forth over PCIe every step would cost 12 seconds, which is longer than the whole computation. Doing the update on the CPU avoids the round trip entirely, which is why the second version is the one that gets used.

What it costs

The cost is the link between the CPU and the GPU, which is PCIe. It carries roughly 60 GB per second, which is far below NVLink and comparable to a network cable. Offload therefore converts a memory problem into a bandwidth problem. It earns its place in a run whose binding constraint is GPU memory.

Where the three links sit
LinkGB/sCarries
NVLink, inside a node450gradients between GPUs
PCIe, CPU to GPU60offloaded state and gradients
InfiniBand, between nodes50gradients across the cluster

When it is the wrong tool

If the run is already limited by the wire rather than by memory, offload makes it worse: it adds traffic to solve a problem the run did not have. The check is the one from the cost page — if communication is already a large fraction of step time, spend the effort on moving less, not on moving more to a slower place.

NVMe extends the same idea one step further out, and the same reasoning applies with worse numbers. It makes runs possible that would otherwise not fit at all, at a speed that only makes sense when the alternative is not running.

Carry this forward: offload buys GPU memory by spending PCIe bandwidth, and it helps only when memory is the binding constraint.
BACKThe memory ladder