02 / GROUNDWORK

Three kernels or one

Take three vectors and compute (a + b) × c, then scale the result. Written down it is one expression. Run unoptimised, it is three separate kernel launches.

A kernel ends by writing its output to the card's memory, because that is the only place the next kernel can find it. So the intermediate values in the middle of the expression get written out and read straight back in, for no reason other than that a kernel boundary fell between them.

GPU memory — slow, far from the lanes GPU memory kernel 1 add a b t₁ kernel 2 multiply t₁ c t₂ kernel 3 divide t₂ out 6 trips to memory
Each launch reads its inputs from memory and writes its output back. t₁ and t₂ make a round trip apiece for nothing.

The arithmetic is not the cost

One add, one multiply, one divide per element. Twenty-one thousand lanes dispatch that almost instantly. What they cannot skip is the waiting. Memory sits far from the lanes and runs far slower than they do, so the cores idle while numbers travel back and forth. The expense of this sequence is the six trips, not the three operations.

Fusion

Do all three inside one kernel. Read a, b and c once, hold the intermediates in registers sitting right beside the lane, write the answer once. Two trips instead of six, for identical arithmetic.

GPU memory GPU memory one fused kernel add multiply divide t₁ t₂ intermediates stay in registers a b c out 2 trips to memory
Registers sit next to the lane, so the hops between stages cost almost nothing. Only the true inputs and the final output touch memory.
6trips, three kernels
→
2trips, fused
 
0change in arithmetic

Drawing the boundary yourself

Nothing about the hardware forced three launches. The boundaries came from the fact that add, multiply and divide are three library functions, each written to be a complete kernel on its own. Call them in sequence and you get their boundaries whether you wanted them or not.

A CUDA kernel is where you draw the boundary instead. You write one function that takes a, b and c, does the whole expression for one element, and returns the answer. The intermediates never become tensors. They are values in a register that exist for a few instructions and are gone. Three stages collapse into one.

What "one kernel" actually costs you

The fused version is faster and it is also less flexible. Each library op was general: any shape, any dtype, composable with anything. The fused kernel does one specific sequence. That trade — generality for trips to memory — is the whole reason torch.compile exists, since it tries to find the fusions automatically instead of making you hand-write them.

This single move explains a great deal of production deep learning tooling. torch.compile looks for chains of small operations and welds them together. FlashAttention is the same idea applied to the attention computation, which is otherwise a parade of large intermediates written out and read back. Fused optimizers do it for the update step. Different problems, one technique: stop bouncing intermediates off memory.

The same lesson, one order of magnitude out

Inside one card, an unfused kernel wastes trips to local memory, and that is already enough to dominate the runtime. Between cards, every exchange crosses a network that is slower still by a wide margin. Once we start splitting a model across several GPUs, the question on every page will be the one this page asks: how many times does data have to move, and can we move it less?

BACKWhat a GPU is