13 / THE ARC

Precision and what Blackwell changes

Blackwell is the current generation of NVIDIA hardware, following the generation the H100 belongs to. The B200 and GB200 are Blackwell cards. Their arithmetic units can multiply 8-bit floating point numbers directly in hardware, which the previous generation could not do at the same granularity.

An 8-bit number holds far less detail than a 16-bit one, so it cannot be used naively. The technique that makes it work is to store a shared scale factor alongside a small block of numbers. MXFP8 is the Blackwell-native form of this, using blocks of 32 values with one shared 8-bit exponent.

one MXFP8 block — 32 values, one shared scale 32 values, one byte each scale 1 byte 33 bytes for 32 numbers — the scale costs 1/32 of a byte per value across the weight and the gradient that is 2 × 1/32 = 0.0625 bytes per parameter
The shared exponent is what lets eight bits carry a usable range. It is not free, but it is nearly free.

What it does to the sixteen bytes

The effect on this session's arithmetic is smaller than it first appears.

Per-weight footprint under each precision
Bytes per weight30B model
16-bit weights and gradients16.00447.0 GiB
8-bit weights and gradients14.06392.9 GiB

Moving the weights and gradients to 8 bits removes two of the sixteen bytes and adds back a small amount. Each block of 32 values carries its own shared scale byte, which is 0.0625 bytes for every parameter across the two tensors. The 32-bit copy and the two running averages account for twelve of the sixteen and are unaffected, because the update arithmetic still needs the accuracy. The reduction in stored state is 12.1 percent.

The gains from 8-bit arithmetic appear in three other places. Matrix multiplication runs faster, activation memory falls, and the volume crossing the interconnect falls with it. TorchTitan reports pre-training up to 41 percent faster for a large mixture-of-experts model on B200 using MXFP8, and loss curves over 1,500 steps that match 16-bit training.

Where 8 bits actually pays
EffectSize of it
stored training state16.00 → 14.06 bytes per weight, a 12.1% cut
matrix multiplicationthe arithmetic units run 8-bit natively, so the maths itself is faster
activation memorythe intermediate values are stored in the narrower format too
interconnect volumeP halves, so every multiple of P halves with it

Only the first of those is a memory saving on the training state, and it is the smallest of the four. The other three are the reason the format is worth adopting at all.

What stays at higher precision

One part of the model is left at higher precision. The attention softmax amplifies small errors in its input, because the exponential turns a modest gap between two scores into a very large ratio. Keeping it in 32-bit costs almost nothing, because softmax is limited by memory bandwidth and not by arithmetic. The operations held at high precision are the operations where high precision is close to free.

two attention scores, and the same two after exp() scores 8.00 8.25 a gap of 0.25, the kind 8-bit rounding creates after exp 2981 3828 a 28% difference in the weight assigned the exponential is what makes this one operation unsafe to narrow
A rounding error the other operations would absorb becomes a visible change in which token gets attended to.

The same argument as the kernel page

Three pages in, the point was that an unfused chain of small operations spends its time waiting for memory rather than computing. Softmax is exactly that kind of operation: it reads a row of scores, does one cheap pass over them, and writes a row back. The lanes are idle either way, so widening the arithmetic from 8 bits to 32 changes the runtime very little — it was never the arithmetic that was slow.

Which gives the rule its shape. Narrow the operations that are limited by arithmetic, because that is where the speed is; leave the ones limited by memory alone, because narrowing them buys nothing and costs accuracy.

Reading the 41 percent

That figure is a reported result for one mixture-of-experts model on one platform, not a general speedup. It is quoted here because the matching loss curves are the load-bearing part: a faster step is only worth having if the model still learns the same thing, and 1,500 steps of agreement is evidence for that rather than for the percentage.

16-bit 2 2 4 8 16.00 8-bit 1 1 4 8 14.06 1.94 the copper sliver at 0.0625 bytes is the shared scale — the twelve bytes on the right do not move
Halving the two leftmost bands recovers 1.94 bytes. The optimizer bands are three quarters of the row and are untouched.

Why the saving is modest

This is the same lopsidedness the first page identified, seen from the other side. The optimizer state is twelve of the sixteen bytes, so halving the other four can only ever recover about a tenth of the total. Precision is a large win for arithmetic speed and a small one for stored state, which is the opposite of what the phrase "8-bit training" suggests.

Try the two levers

Two independent choices govern the footprint: what precision the matrix multiply runs in, and what precision Adam keeps its moments in. One square is one byte.

MATMUL PRECISION
ADAM MOMENTS
flip either toggle — the dashed outline holds the bytes removed
BYTES OF PERSISTENT TRAINING STATE PER PARAMETER, ONE SQUARE PER BYTE
WHAT THE CHOICE CHANGES, ON ONE FIXED 0 TO 100 PERCENT AXIS

Where it does matter

The communication figures move with the parameter format, and there the four bytes are the whole story. P is one copy of the parameters, so 8-bit parameters halve P from 60 GB to 30 GB, and every multiple of P on the cost page halves with it.

P and the traffic it measures
Parameter formatP2P over InfiniBand3P over InfiniBand
16-bit60 GB2.40 s3.60 s
8-bit30 GB1.20 s1.80 s

That is a much larger effect than 12 percent, and it lands on the constraint the cost page showed getting worse each hardware generation. Blackwell shortens the compute, which raises communication as a share of the step; 8-bit parameters cut the communication back. The two changes arrived together for a reason.

Carry this forward: 8-bit precision takes about a tenth off stored state and about a half off communication, because the optimizer dominates memory and the parameters dominate the wire.
BACKOverlapping communication