Precision and what Blackwell changes
Blackwell is the current generation of NVIDIA hardware, following the generation the H100 belongs to. The B200 and GB200 are Blackwell cards. Their arithmetic units can multiply 8-bit floating point numbers directly in hardware, which the previous generation could not do at the same granularity.
An 8-bit number holds far less detail than a 16-bit one, so it cannot be used naively. The technique that makes it work is to store a shared scale factor alongside a small block of numbers. MXFP8 is the Blackwell-native form of this, using blocks of 32 values with one shared 8-bit exponent.
What it does to the sixteen bytes
The effect on this session's arithmetic is smaller than it first appears.
| Bytes per weight | 30B model | |
|---|---|---|
| 16-bit weights and gradients | 16.00 | 447.0 GiB |
| 8-bit weights and gradients | 14.06 | 392.9 GiB |
Moving the weights and gradients to 8 bits removes two of the sixteen bytes and adds back a small amount. Each block of 32 values carries its own shared scale byte, which is 0.0625 bytes for every parameter across the two tensors. The 32-bit copy and the two running averages account for twelve of the sixteen and are unaffected, because the update arithmetic still needs the accuracy. The reduction in stored state is 12.1 percent.
The gains from 8-bit arithmetic appear in three other places. Matrix multiplication runs faster, activation memory falls, and the volume crossing the interconnect falls with it. TorchTitan reports pre-training up to 41 percent faster for a large mixture-of-experts model on B200 using MXFP8, and loss curves over 1,500 steps that match 16-bit training.
| Effect | Size of it |
|---|---|
| stored training state | 16.00 → 14.06 bytes per weight, a 12.1% cut |
| matrix multiplication | the arithmetic units run 8-bit natively, so the maths itself is faster |
| activation memory | the intermediate values are stored in the narrower format too |
| interconnect volume | P halves, so every multiple of P halves with it |
Only the first of those is a memory saving on the training state, and it is the smallest of the four. The other three are the reason the format is worth adopting at all.
What stays at higher precision
One part of the model is left at higher precision. The attention softmax amplifies small errors in its input, because the exponential turns a modest gap between two scores into a very large ratio. Keeping it in 32-bit costs almost nothing, because softmax is limited by memory bandwidth and not by arithmetic. The operations held at high precision are the operations where high precision is close to free.
The same argument as the kernel page
Three pages in, the point was that an unfused chain of small operations spends its time waiting for memory rather than computing. Softmax is exactly that kind of operation: it reads a row of scores, does one cheap pass over them, and writes a row back. The lanes are idle either way, so widening the arithmetic from 8 bits to 32 changes the runtime very little — it was never the arithmetic that was slow.
Which gives the rule its shape. Narrow the operations that are limited by arithmetic, because that is where the speed is; leave the ones limited by memory alone, because narrowing them buys nothing and costs accuracy.
Reading the 41 percent
That figure is a reported result for one mixture-of-experts model on one platform, not a general speedup. It is quoted here because the matching loss curves are the load-bearing part: a faster step is only worth having if the model still learns the same thing, and 1,500 steps of agreement is evidence for that rather than for the percentage.
Why the saving is modest
This is the same lopsidedness the first page identified, seen from the other side. The optimizer state is twelve of the sixteen bytes, so halving the other four can only ever recover about a tenth of the total. Precision is a large win for arithmetic speed and a small one for stored state, which is the opposite of what the phrase "8-bit training" suggests.
Try the two levers
Two independent choices govern the footprint: what precision the matrix multiply runs in, and what precision Adam keeps its moments in. One square is one byte.
Where it does matter
The communication figures move with the parameter format, and there the four bytes are the whole story. P is one copy of the parameters, so 8-bit parameters halve P from 60 GB to 30 GB, and every multiple of P on the cost page halves with it.
| Parameter format | P | 2P over InfiniBand | 3P over InfiniBand |
|---|---|---|---|
| 16-bit | 60 GB | 2.40 s | 3.60 s |
| 8-bit | 30 GB | 1.20 s | 1.80 s |
That is a much larger effect than 12 percent, and it lands on the constraint the cost page showed getting worse each hardware generation. Blackwell shortens the compute, which raises communication as a share of the step; 8-bit parameters cut the communication back. The two changes arrived together for a reason.