What a GPU is
Four multiplications, sitting side by side. None of them needs the answer to any other.
Give this to a single CPU thread and it takes four cycles, because one worker does one multiply at a time and there is no way around that. Give it to a GPU and it takes one, because there are four workers and each takes a row.
The total work is identical. Four multiplications happen either way. What changed is
wall-clock time, and the only reason it could change is that no row needed a result from
another row. Independence is what buys the speedup. Had row two been
R₁ × 9, the GPU would be back to four cycles as well.
Single instruction, multiple data
The name for this arrangement is SIMD. One instruction is broadcast to many lanes, and
each lane applies it to its own data. Here the instruction is mult. The four
pairs are the multiple data.
The unit you actually write is a kernel — the smallest block of computation you can launch on a GPU. A multiply kernel is about as simple as they get. Two vectors in, one vector out, one step.
| a | b | a × b |
|---|---|---|
| 4 | 6 | 24 |
| 5 | 9 | 45 |
| 6 | 17 | 102 |
| 4 | 2 | 8 |
A core can run multiplies, adds, subtracts, and fused multiply-adds — the full menu. But under SIMD the lanes in a group do not run different instructions at the same instant. At any moment every lane is executing the same instruction on different data. If one lane needs a different instruction than its neighbours, the group splits and runs the branches one after the other. That is the dependency problem again, wearing a different hat.
The scale of it
Four lanes made the point. Real hardware is not four.
Specification for a GeForce RTX 5090, quoted here only for the order of magnitude.
Twenty-one thousand lanes, all waiting for work that is independent enough to fill them. Neural network math obliges. A matrix multiply is nothing but an enormous pile of small independent products, summed into place — and multiply-then-accumulate is itself a single fused instruction on this hardware. The math and the machine found each other.
Why this page is here
The four lanes above assumed all four input pairs were sitting in the same memory, reachable for free. Inside one card that holds: the lanes share memory and talking is nearly free. Once the workers are on four different cards, every exchange between them travels over a wire and costs real time. Free inside, expensive between — that gap is what every scheme in this session is negotiating with.