Optimization worksheet · sheet 1 of n

Three steps, forever:

1. FORWARD    push all N samples through the model    → predictions ŷ
2. LOSS       compare ŷ to y                          → one number, L
3. BACKWARD   ask each parameter: "if I nudge you up by 1, how much does L change?"
              → that answer is ∂L/∂θᵢ, one number per parameter
4. UPDATE     θᵢ ← θᵢ − η·∂L/∂θᵢ                 → move every parameter against its own slope

Nothing in step 4 knows about step 4 of any other parameter. Each one moves against its own partial derivative, and they all move in the same instant. That independence is what makes it cheap, and it is also the crack that widens later on this page.

The data, fixed for the whole sheet

Two parameters this time: a weight w and a bias b. The data comes from y = 3x, so the answer is w* = 3, b* = 0. We start both at zero.

L(w,b) = (1/2N) Σ (w·xₕ + b − yₕ)²

∂L/∂w = (1/N) Σ (w·xₕ + b − yₕ)·xₕ      ← residual weighted by the input
∂L/∂b = (1/N) Σ (w·xₕ + b − yₕ)            ← plain average residual

Before the three-row dataset, the smallest case that still shows everything. One weight starting at zero, and a loss written directly as a function of that weight:

L = (w − 5)²        w₀ = 0

L(0) = 25       ← large, as expected
L(5) = 0        ← the answer, readable by eye

With one parameter you can just look at it and know the answer is 5. With three you can still solve it. With a billion you cannot look at it at all, which is the entire reason we differentiate instead of solving:

∂L/∂w = 2(w − 5)        ← the 2 falls out of differentiating the square

At w = 0:   ∂L/∂w = 2(0 − 5) = −10

Substituting into the update rule

Write d = w − 5 for the distance still to travel. Put the gradient into w ← w − ηg and subtract 5 from both sides:

wₙₑₖ      = w − η·2(w − 5)
wₙₑₖ − 5  = (w − 5) − 2η(w − 5)
dₙₑₖ      = d·(1 − 2η)                    ← (w−5) factors out completely

The whole run is now one number. The remaining distance is multiplied by 1 − 2η every single step, and that factor never changes. It carries two pieces of information at once:

convergence requires   |1 − 2η| < 1   ⇔   0 < η < 1        for this loss

Four behaviours from one number

At 0.01 the weight crawls and will need several hundred steps. At 0.10 it converges cleanly. At 0.90 it overshoots the target on every step, but the overshoot shrinks each time, so it still arrives. At 1.10 the overshoot grows instead and the run is lost.

Compare the middle two rows. Both have multiplier magnitude 0.80, so both close the same fraction of the distance per step, and their loss curves are identical — 0.90 simply spends every step on the far side of the minimum. Sign and magnitude are answering two different questions, and only magnitude decides whether training works.

η = 0.1, factor 0.8, five steps

step 1:  g = 2(0 − 5) = −10       w = 0 + 1.0    = 1.000    d = −4.000  = −5 × 0.8 ✅
step 2:  g = 2(1 − 5) = −8        w = 1.0 + 0.8  = 1.800    d = −3.200
step 3:  g = 2(1.8 − 5) = −6.4    w = 1.8 + 0.64 = 2.440    d = −2.560

Every regime of η, on one loss

Read the η = 1.0 row carefully. The weight lands on 0 and 10 alternately and the loss is 25 at every step forever. Gradient descent is computing a correct downhill direction on every iteration and making zero progress. The direction was never the problem.

And η = 0.5 hits the answer in one step because ∂²L/∂w² = 2, so 1/curvature = 0.5. Sheet 2 shows that this is the general rule, and why you can never actually use it.

The three lines of algebra in row 2 did not depend on the number 5, or on the 2. They work on any loss of that shape. If the gradient comes out as c·d instead of 2d, the multiplier is:

dₙₑₖ = d(1 − ηc)          convergence needs |1 − ηc| < 1   ⇔   0 < η < 2/c

That c is the curvature: how sharply the loss bends around its minimum, formally ∂²L/∂w². It was 2 in row 2, which is where 0 < η < 1 came from. Two consequences, both worth stating plainly:

A real loss has more than one curvature

It bends sharply along some directions and gently along others, and one η has to serve all of them at once. Take two directions in one loss:

L = ½(r·u² + v²)          ∂L/∂u = r·u          ∂L/∂v = v

curvature along u = r        curvature along v = 1

starting from (u, v) = (1, 1), each step multiplies
    u by (1 − ηr)      and      v by (1 − η)      ← row 2's algebra, twice

Two independent copies of the same recursion, sharing one η. Drag r and watch the gap open:

The same readings, written out, so the shape of the cost is visible without dragging:

Doubling the curvature ratio halves the permitted step and roughly doubles the wait. The cost is linear in r, and r in a transformer is not 50.

Sitting at the cap, r = 20, η = 0.100

u multiplier = 1 − (0.1)(20) = −1.000        v multiplier = 1 − 0.1 = 0.900

At the cap the steep direction bounces between +1 and −1 and never improves, while the shallow direction inches down by 10% a step. Back η off to 0.05 and u lands exactly on zero in one step, but then v needs 45 steps to close 90% of its distance. There is no value of η that serves both well, and that is with a ratio of only 20.

Gradient descent in two dimensions

Fix r = 20 and move η instead. Both coordinates use the same learning rate; u reacts twenty times harder to it, because the curvature along u is twenty times larger.

Row 8 measures this on the actual two-parameter fit from row 4, where the ratio comes out to 46 without anyone choosing it.

Step 1 — forward at w = 0, b = 0

Step 2 — loss

L = 126 / (2 × 3) = 21.000

Step 3 — both gradients, straight off the Σ row

∂L/∂w = (1/3)(−42) = −14.000     ← the column Σ(r·x), divided by N
∂L/∂b = (1/3)(−18) = −6.000      ← the column Σ(r),   divided by N

Read the signs. Both negative, so both parameters are too small, and w is wrong by more than b is. No magic: the gradient of a squared error is just the residual, re-weighted by whatever multiplied that parameter in the forward pass. For w that multiplier was x. For b it was 1.

Step 4 — update, η = 0.1

w ← 0 − (0.1)(−14.000) = 1.400
b ← 0 − (0.1)(−6.000)  = 0.600

Iteration 2, from scratch with the new parameters

L       = 25.400 / 6      = 4.2333      ← was 21.000
∂L/∂w  = (1/3)(−18.800) = −6.2667     ← was −14.000, the slope flattens as we approach
∂L/∂b  = (1/3)(−7.800)  = −2.6000

w ← 1.400 + 0.62667 = 2.02667
b ← 0.600 + 0.26000 = 0.86000

The loss drops by 250× in five iterations, and then almost stops. Look at the last column: ∂L/∂b has flipped sign. The bias overshot to 1.021 when the answer is 0, and now it has to come back — slowly, because both gradients are now small.

Iterations 5 through 400 exist entirely to walk the bias back from 1.02 to 0.009. The algorithm is working correctly the whole time. It is just crawling along a direction where the loss barely changes, and row 8 explains why that direction exists.

Every gradient above was averaged over all three rows before a single parameter moved. That is the defining property, and it has a price:

On the last row a single parameter update would take weeks. This is the first, and least interesting, reason gradient descent is not what trains a language model — you replace the full-batch average with a mini-batch estimate and get SGD. The deeper reasons are about the shape of the loss, not the size of the data.

Take a 1D convolution with a two-tap kernel [k₁, k₂] slid over an input of length 4. The same two numbers produce all three outputs:

x = [1, 2, 3, 4]        target t = [3, 5, 7]        (true kernel: k₁=1, k₂=1)

o₁ = k₁(1) + k₂(2)
o₂ = k₁(2) + k₂(3)          ← k₁ appears in all three, k₂ appears in all three
o₃ = k₁(3) + k₂(4)

At k₁ = k₂ = 0, residuals are [−3, −5, −7]:

∂L/∂k₁ = (1/3)[(−3)(1) + (−5)(2) + (−7)(3)] = (1/3)(−34) = −11.333
∂L/∂k₂ = (1/3)[(−3)(2) + (−5)(3) + (−7)(4)] = (1/3)(−49) = −16.333

Three properties fall out of that arithmetic, and all three are what gradient descent wants:

Scale that up to a real filter. A 3×3×32 kernel is 288 parameters that all watch the same patch of the same image, all fire on the same edges and textures, and all receive gradient from every one of the tens of thousands of spatial positions in the batch. They are not 288 independent problems. They are one problem with 288 coordinates that happen to point in similar directions with similar curvature.

A transformer is mostly nn.Linear and embedding tables. Nothing is shared across positions the way a kernel is. Three failures, each with its arithmetic.

Failure 1 — most parameters get no gradient at all

An embedding table is a lookup. A row that is not looked up in this batch gets a partial derivative of exactly zero:

Roughly 270× separates the two rows that did move. Whatever η is correct for "the" is 270× too large or too small for "model". In a convolution this situation cannot arise, because there is no row to skip.

Failure 2 — one step size, many curvatures

This is visible in our own two-parameter example. Write out the second-derivative matrix for L(w,b):

H = (1/N) [ Σx²   Σx ]  = (1/3)[ 14   6 ]  = [ 4.6667   2.0000 ]
          [ Σx    N  ]        [  6   3 ]    [ 2.0000   1.0000 ]

eigenvalues:   λ₁ = 5.5465        λ₂ = 0.1202
condition number κ = λ₁/λ₂ = 46.1

Those two eigenvalues are two directions in (w,b) space with wildly different steepness, and each one imposes its own demand on η:

η must satisfy   η < 0.361   (or the steep direction explodes)
η should be      η = 8.319   (or the flat direction crawls)

ratio of what we need to what we are allowed:  8.319 / 0.361 = 23×   ❌

We are forced to pick 0.1, which is 46× too small for the flat direction. That is the 400-iteration crawl in row 5, and it came from two parameters with a condition number of 46. Transformer loss surfaces run condition numbers in the thousands to millions, across billions of coordinates.

Failure 3 — the parameters are learning different things

Put it as a question about linkage. If a set of parameters is linked to each other, they can all travel in the same direction at the same speed, and one shared step size describes that motion honestly. If they are not linked, each one needs its own movement, and a single η is a statement about all of them that is true of none of them.

Weight sharing forced the conv kernel's coordinates to be alike. A linear stack has the opposite property: one direction in the weight space ends up carrying arithmetic, another syntax, another the behaviour of a token that shows up in one batch in a thousand. They are near-independent problems sharing a loss and a step size. They have no reason to want the same η, and gradient descent gives them no way to ask for a different one.

Both columns run the identical update rule. The difference is not in the mathematics of gradient descent, it is in whether the parameters it is applied to resemble each other. Weight sharing manufactures that resemblance. A linear layer does not.

A weight whose gradient has kept the same sign for many steps can safely be moved further than one whose gradient alternates. Using that fact needs a record of recent gradients, and the record has to have a fixed size no matter how long training runs.

An exponential moving average does exactly that. One running number, and each new observation is mixed into it by a fixed fraction:

m ← β₁·m + (1 − β₁)·g          β₁ = 0.9 in almost every setup

Each new gradient contributes one tenth; the existing average keeps nine tenths. A gradient's share is multiplied by β₁ again on every later step, so it fades:

After roughly 1/(1 − β₁) = 10 steps a gradient has faded to almost nothing. That is the sense in which m remembers the last ten gradients while storing one number, and the memory costs one value per parameter rather than one list per parameter.

The two directions produce two characteristic gradient shapes

Before the averaging, look at what is being averaged. Drag η and read the gradient each direction hands back, scaled by the first gradient so the two rows are comparable:

Row 3's loss gives both of them at once. Along the steep direction the sign flips on every step, because the weight lands past the minimum each time. Along the shallow direction the sign never changes, because the weight approaches from one side and never arrives. Feed each into the same EMA, with unit gradients to keep the arithmetic readable:

steep:    g = +1, −1, +1, −1, ...
  m₁ = 0.9(0)      + 0.1(+1) = +0.1000
  m₂ = 0.9(0.1)    + 0.1(−1) = −0.0100      ← the two nearly cancel
  m₃ = 0.9(−0.01) + 0.1(+1) = +0.0910

shallow:  g = +1, +1, +1, +1, ...
  m₁ = 0.9(0)      + 0.1(1) = 0.1000
  m₂ = 0.9(0.1)    + 0.1(1) = 0.1900
  m₃ = 0.9(0.19)   + 0.1(1) = 0.2710      ← the terms add up

Same rule, same β₁, same gradient magnitude of 1. After eight steps the consistent direction has built an average of 0.57 and is still climbing toward 1. The alternating direction is stuck near zero and changes sign every step.

The steady state, which is the whole point

constant g:      m → g                              gain 1.000
alternating g:   m → ±g(1 − β₁)/(1 + β₁) = ±0.1/1.9   gain 0.0526

ratio = 1 / 0.0526 = 19×

Substituting m for g in the update, w ← w − ηm, the consistent direction moves at full speed while the oscillating one is throttled to a nineteenth. Nothing was measured, nothing was tuned per parameter, and no second learning rate was introduced. The cancellation does the work, and it is free because the two shapes are already in the gradients.

One β, both halves

Momentum fixed the oscillation and left every weight sharing one η. Look back at row 3: we wrote 1 − 20η for one direction and 1 − η for the other. The 20 belongs to the loss and cannot be edited. The η is the only thing we control, and changing it changes it for every parameter at once.

In a language model the weights differ enormously in how often they receive a gradient at all. The embedding row for a common word gets a large gradient in nearly every batch; the row for a rare one may get nothing across a thousand batches. This does not happen in image training, where a dataset holds ten thousand cats and ten thousand dogs and no class is ever absent. Here the model is routinely asked about a concept it saw twice.

A second average, this time over the squared gradient

v ← β₂·v + (1 − β₂)·g²          β₂ = 0.999

1/(1 − β₂) = 1000        ← this average spans about a thousand gradients, not ten

if g holds steady at some value:   v → g²   and   √v → |g|

The subscript separates it from the β₁ of row 10. That average tracked which way the gradient points; this one tracks how big it usually is. Divide the step by it:

w ← w − (η / √v)·g

Set the two updates next to each other. The only edit is what sits in front of g:

wₙ = w₀ − η·g              one η, shared by every parameter in the model
wₙ = w₀ − (η/√v)·g        η still shared, but √v belongs to this parameter alone

At steady state, where each parameter's gradient has held at its own typical size long enough for v to settle on g²:

B is handed a learning rate a hundred times larger than A's, and the two then take exactly the same step. The division strips the gradient's magnitude out of the update and keeps only its sign and its consistency. A parameter that has been inactive for a thousand steps therefore takes a full-sized step the moment a gradient arrives, which is exactly what a rare word's embedding needs.

Two arbitrary values, worked out loud

Forget where they came from for a moment. Pick any two values of √v, one small and one a little above 1, and give them to two weights:

w₁:  √v = 0.01                w₂:  √v = 1.02

√v sits in the denominator, so the factor each weight receives is its reciprocal:

Follow the chain backwards and the whole mechanism is one inverse proportion. v is small because g² was small; g² was small because that weight has barely been touched by the loss; 1/√v is therefore large; so the weight gets a large learning rate without anyone deciding it should. Big gradient history divides the rate down, small gradient history multiplies it up, automatically, on every step, for every parameter.

Why 1 − β₂ is so small

Write the recursion with the constants substituted in and the weighting is obvious:

vₙ = 0.999·v₀ + 0.001·g²

How large must a new gradient be to move v by 10 percent?
    0.001(g² − v) = 0.1v      →   g² = 101v      →   |g| ≈ 10× the usual size

A single unusual batch barely registers. Only a gradient whose square is a hundred times the running average shifts v by a tenth, and to move it substantially the new size has to persist across many steps. That is deliberate: after a thousand steps v is a signal lock on this parameter's typical scale, and one noisy batch should not be allowed to break it.

This is unrelated to gradient clipping, which is worth separating since the two are often confused. Clipping rescales a whole gradient vector whose norm exceeds a threshold, before the optimizer sees it, and acts on a single step. v acts on the accumulated scale over a thousand steps, per parameter, and never discards anything.

The quantity η/√v is a learning rate belonging to that parameter alone, and that is what names this row. Away from steady state the same thing holds with the numbers moving: two parameters, one receiving g = 1.00 on every step and one receiving g = 0.01, after two hundred steps with η = 0.010:

Gradients a hundred times apart, learning rates a hundred times apart in the opposite direction, and identical steps. The quiet parameter is handed a rate 100× larger precisely because its gradients are small, which is the transcript's rule turned into arithmetic: small amplitude, bigger multiplier.

The parameter that receives nothing

A rare row is worse off than merely quiet. On the batches where its token is absent the gradient is not missing, it is computed and comes back essentially zero, because nothing in the batch connects that row to the loss. Watch what v does across the silence:

batch 1:      g = 1.0    →  v = 0.001·(1.0)² = 0.001000
batches 2–1000: g ≈ 0    →  v ← 0.999·v each time, 999 times

v = 0.001 × 0.999⁹⁹⁹ = 0.000368       √v = 0.01918
η/√v = 0.010 / 0.01918 = 0.5212          ← against 0.0235 for the loud parameter

Its own learning rate has climbed 22× while it waited. When the token finally appears again, that one gradient moves the row 22 times further than the same gradient would move a common row. The silence itself is the signal that this parameter needs a bigger multiplier.

Momentum covers the other half of the same problem. On the silent batches m still holds the direction from the last real gradient, so the row can keep moving on the strength of its history rather than sitting frozen. This is dead reckoning: when the satellite fix drops out, an aircraft does not stop navigating, it carries forward the last known velocity and its acceleration until a new fix arrives. A neural network parameter in a sparse batch is in exactly that position.

What it costs

Two running numbers per parameter, m and v, both the same size as the parameter itself:

Following the direction was not enough; the scale had to be followed too, and each parameter has to be tracked twice. That is a large part of why optimizer state, not the model, sets the memory bill for training.

Two questions worth pinning down

Is "amplitude" the same thing as η? It is the step length, which under a plain update is η times the gradient and under this row is (η/√v)·g. Raising a parameter's amplitude means giving it a larger η/√v, which is a per-parameter learning rate rather than the global η. The global η never changes between parameters; only the divisor does.

What does "acceleration" mean here? Take derivatives of the loss and count them:

The gradient says how fast the loss changes as the weight moves. The rate at which that gradient itself changes is the curvature, and that is the second-order term. It is what set the cap 2/c in row 3, so it has been the governing quantity all along.

Two things are worth separating, though. The true second derivative for a model with n parameters is an n×n matrix, so for a billion parameters it has 10¹⁸ entries and cannot be formed, let alone inverted. What an optimizer actually stores is v, an average of g², which is one number per parameter and is not the second derivative. It works as a stand-in for a reason visible in row 3's own algebra:

for a quadratic:   g = c·d          √v → |g| = c·|d|

step = (η/√v)·g = η·(c·d)/(c·|d|) = η·sign(d)

The curvature c cancels. Dividing by √v removes exactly the factor that made one direction unusable at the other's learning rate, which is why a per-parameter method survives a condition number of 46 or of 10⁶ without ever computing a second derivative.

η is a base number with multipliers stacked in front of it

Nothing in this row or the last edits η. It is set once, typically at 0.010 or 0.0003, and every mechanism we have added sits in front of it as a factor. Reading the update as a product makes what each piece does visible:

step = η  ×  sₖ  ×  1/√v  ×  m

        η      the base rate, one number for the whole model, chosen once
        sₖ     the schedule, one number per step, same for every parameter
        1/√v  the per-parameter factor, different for every weight, from its own history
        m      the direction, an average of recent gradients rather than the current one

One step of training, η = 0.010, with the schedule at full value and then late in the run:

Same η on every line. The third column is the only one that distinguishes parameters, and it is computed rather than chosen. The fourth line shows the schedule acting on the whole model at once, which is why decaying η cannot substitute for a per-parameter factor and a per-parameter factor cannot substitute for the schedule.

Row 10 built an average of the gradient, which sets the direction. Row 11 built an average of the squared gradient, which sets the scale. Adam is those two, nothing more:

m ← β₁m + (1 − β₁)g          β₁ = 0.9      direction, remembers ~10 steps
v ← β₂v + (1 − β₂)g²         β₂ = 0.999    scale,     remembers ~1000 steps

w ← w − η·m̂ / (√v̂ + ε)          ε = 10⁻⁸

ε exists only so the division cannot blow up when v is zero or nearly zero, which it is on the first step of training and on any parameter that has never received a gradient.

Why the hats: both averages start at zero

At t = 1 with β₂ = 0.999, v is one thousandth of g² purely because it began at zero and has had one observation. Dividing by √v then produces a step far larger than intended. The correction divides out exactly the fraction of the average that has actually accumulated:

m̂ = m / (1 − β₁ᵗ)          v̂ = v / (1 − β₂ᵗ)

Take a parameter whose gradient holds steady at g = 0.1, and read the step in multiples of η:

With correction the step is exactly η from the first iteration, whatever the gradient's size. Without it the first steps are three to six times too large, which is a reliable way to destroy a model in its first hundred iterations. Notice also that m̂ lands on 0.1 and v̂ on 0.01 immediately: the hats make the averages read as the quantities they estimate rather than as fractions of them.

Bias correction is exact, not a patch

Feed a constant gradient g into the first average and after t steps it has reached mₜ = g(1 − β₁ᵗ), short of g by exactly that factor. Dividing by it restores the value, and the same argument gives v. Nothing is approximated.

at t = 1:   m₁ = 0.1g          v₁ = 0.001g²

without correction:   m₁/√v₁ = 0.1g / (0.0316|g|) = 3.16      ← whatever g was
with correction:      m̂₁/√v̂₁ = g / |g|            = 1.00

The mismatch exists because β₂ = 0.999 sits much closer to 1 than β₁ = 0.9, so the two averages are pulled toward zero by very different amounts. Correcting both makes the factors cancel.

Five gradients, one weight, every intermediate number

The step column is the one to watch. With correction on, gradients ranging from 0.40 to 0.60 produce steps that all sit within half a percent of 0.001, the learning rate itself. Switch the preset to gradients a hundred times smaller and that column is identical to the digit. Turn correction off and the same five gradients produce 3.16η rising to 5.78η.

The objection: how can it know a word is rare before training ends?

It cannot, and it never tries to. v is a running statistic over roughly the last thousand steps, not a property of the dataset. The reasoning is local: this parameter's gradients have been small lately, so take a larger step on faith. If that step was too large, the very next gradient says so, and it says so with a bigger g² which raises v and shrinks the next step automatically. The correction is continuous and self-repairing.

Which is also why none of this can be judged from a single iteration. Momentum, velocity, acceleration and averaging are statements about hundreds of steps. The first stretch of training is genuinely noisy, the averages are still forming, and that is the reason the learning rate is ramped up rather than applied at full value from step one.

Weights drift upward over a long run, and large weights make a model brittle. The remedy is to add a second term to the loss that charges for size:

Lᵗₒₜₐ⃿ = Lₜₐₛₖ + ½λΣw²          λ typically 0.1 to 0.001

Any loss written as a sum of two terms becomes a negotiation between them, and the constant in front decides who wins. If one term is 1.2 and the other is 10,000, the model optimises the second and ignores the first entirely. λ is what puts the two on comparable footing:

The gradient of the new term is simply λw, so every weight is pulled toward zero in proportion to its own size. A weight that matters to the task is held in place by the task gradient; a weight that contributes nothing has nothing opposing the pull, and it decays away. The same structure appears anywhere two objectives share a loss: style transfer weighting content against style, or a detector weighting one correct box against four background boxes, where the background term must be divided down or the model learns to answer "no object" to everything and scores 80%.

What w² actually means

Written with one weight it looks harmless. It is a sum over every parameter in the model:

½λΣw² = ½λ(w₁² + w₂² + w₃² + … + w₁₀₀₀₀₀₀₀₀₀²)

typical weight magnitude ≈ 0.02      →   w² ≈ 4×10⁻⁴
one billion of them                 →   Σw² ≈ 4×10⁵

Against a task loss of about 2, that term is enormous, which is the point about λ. And it is also why the penalty is not literally added to the loss in a real implementation: the only thing the optimizer ever needs is the derivative, λw, which is one cheap multiply per parameter. The loss form explains the intent, the update form does the work.

The task loss itself is the familiar one, summed over the vocabulary or the batch. Predictions against targets, differences squared, added up:

targets      0.00   1.00   0.00
predictions  0.01   0.70   0.20
residuals   −0.01  +0.30  −0.20
squares      0.0001 0.0900 0.0400      L = 0.1301

Under Adam, the same λ does not mean the same decay

Under plain gradient descent, adding λw to the gradient shrinks every weight by the same fraction each step. Under Adam it does not, because that added λw is divided by √v̂ along with everything else. Take two weights both sitting at 0.5, with λ = 0.1 and η = 0.001:

decoupled amount:  ηλw = 0.001 × 0.1 × 0.5 = 5.0 × 10⁻⁵
L2 route:          that same amount, divided by √v̂

B is decayed a hundred times more strongly than A, for a reason unconnected to how large B is. How much regularization a weight receives has become a function of its gradient history, which is not what anyone asked for. Both weights are 0.5; both should shrink by the same amount.

The interference, and the fix

Adding λw to the gradient means it also enters m and v. The decay is then rescaled by 1/√v̂, so a parameter with small gradients gets a large decay and a parameter with large gradients gets almost none, which is backwards: the decay was meant to depend on w, not on gradient history. AdamW takes the decay out of the gradient and applies it directly to the weight:

Adam:    w ← w − η·m̂/(√v̂ + ε)          with λw folded into g   ❌
AdamW:   w ← w − η·m̂/(√v̂ + ε) − ηλw                       ✅

Two separate terms, so the adaptive machinery handles the task gradient and the decay stays proportional to the weight. If a run uses weight decay at all, this is the version to use.

Both weights from the table above then shrink by 5.0 × 10⁻⁵, as intended, and nothing about their gradient history enters that number.

η and λ are one setting, not two

A further consequence of the decoupled form, established in 2025: the finished weights are an exponential moving average of the updates applied along the way, with a timescale of 1/(ηλ) steps. The decay term pulls each weight toward zero in proportion to itself, which is structurally the same recursion as row 10's average, so the same 1/(1 − β) counting applies.

timescale = 1/(ηλ)

η = 0.0003, λ = 0.1   →   1/(0.0003 × 0.1) = 33,333 steps

Rows one and four use different η and different λ and produce the same averaging window, which is the point: these are not two independent knobs. Their product is the knob. Halving η without touching λ doubles how far back the final model averages, whether or not that was the intention.

What to exclude from decay

Normalization scales and biases are left out. Shrinking a weight matrix reduces how large the layer's output is; shrinking a normalization scale changes what the layer computes, since that parameter is the layer's calibration rather than its magnitude. Pulling it toward zero is not regularization, it is damage.

Per parameter, in a standard mixed-precision setup:

For an 8B-parameter model the choice of optimizer is the choice of machine:

The cheapest option is the one that does not work and the one that works costs double. Following the direction was never the expensive part; following the scale as well, for every parameter, is what put optimizer state at the centre of the hardware bill.

This is a different thing from a data-mixture warmup, which blends a change of distribution. Here the learning rate itself is being ramped.

Row 12 established that Adam moves each weight by approximately η. How good that approximation is depends entirely on how consistent the gradients have been:

noisy case:   m̂ largely cancels across steps,  √v̂ does not
              RMS ratio  = √((1−β₁)/(1+β₁))          = 0.2294
              mean ratio = √((1−β₁)/(1+β₁))·√(2/π)  = 0.1830

So through most of a run the gradients disagree with each other and every weight takes roughly a fifth of η. That is the regime η was tuned for.

The largest update-to-weight ratio a run ever sees

The quantity that matters is not the step but the step relative to the weight it is moving. Three things set it, and all three change over a run:

ratio(t) = η(t) × c(t) / |w(t)|

    η(t)    the schedule — the ramp, if there is one
    c(t)    gradient consistency — 1.000 while gradients agree, falling to a noisy floor
    |w(t)|  weight magnitude — starts at 1/√fan-in and grows

The initialisation makes this precise. Weights are drawn at a scale of 1/√fan-in so a layer's output has about the same size as its input however wide the layer is. A few full-sized steps in a consistent direction move weights by a sizeable fraction of that scale, the layer's output magnitude shifts, and the drift compounds through the depth of the network.

A hundred agreeing steps at full η can move a weight further than its own initial scale, and the wider the layer the worse it gets. Warmup ramps η from near zero across the first couple of percent of the run, which spans those agreeing steps at a reduced rate and gives the averages time to start reporting real statistics.

With warmup handled, the rest of the run has to be shaped. Almost every model published starts η between 10⁻³ and 10⁻⁴, occasionally 3× or 4× higher. That number is not a maximum or a minimum, it is a baseline; the optimizer scales it per parameter, and the schedule scales it per step. Three shapes are in use:

The last row is the counter-intuitive one. Watching loss fall from 5 to 2 looks like progress and is mostly free; the move from 1.10 to 1.05 is the expensive part, and it happens at small η late in the run.

What you actually supply: a start rate and an end rate

Adam, the moments, the per-parameter divisions are all internal. From outside a run you hand over two numbers and a step count, and the schedule fills in everything between them:

SLR   start learning rate       e.g. 0.001
ELR   end learning rate         e.g. 0.00001
S     total steps               e.g. 1,000,000

The simplest choice is to supply only one number and hold it for all million steps. It fails, and for a reason that has nothing to do with speed. Row 12 established that Adam moves each weight by about η. So η is not merely the pace, it is the resolution — the smallest change the optimizer is able to make.

a weight sitting at   0.0002376
needs to become       0.0002575
required change       0.0000199

at η = 0.001    the smallest available move is ~0.001    →   50× too coarse   ❌
at η = 0.00002  the smallest available move is ~0.00002  →   the right size    ✅

A constant learning rate can carry a weight into the right neighbourhood and can never place it. The final adjustments are physically unavailable until η has come down, which is the same observation as the earlier one about the run from 1.10 to 1.05: not that late training is slow, but that it is the only stretch where fine changes exist at all.

The two ways down

With SLR = 0.001, ELR = 0.00001 over a million steps:

cosine:   η(t) = ELR + (SLR − ELR)·½(1 + cos(πt/S))
linear:   η(t) = SLR + (ELR − SLR)·(t/S)

They meet at the halfway mark and disagree at the quarters. Cosine holds η high for longer at the start, then falls faster through the last quarter, so more of the run is spent covering distance and the fine-placement phase arrives late and quickly. Linear spends more of the run at intermediate rates. WSD is the extreme version of the same preference: hold high, then drop.

WSD is uncomfortable because for most of the run its loss curve sits above the cosine curve, and the payoff only arrives at the drop. The intuition is that a long stretch at high η keeps every parameter agitated, so the run settles into a wide minimum rather than a narrow one, and wide minima generalise better. Two decisions follow from choosing it: where to place the drop, and whether the team can watch a flat loss curve for eighty percent of a run without intervening.

Cosine's structural constraint, and what WSD buys instead

Cosine is defined in terms of the total number of steps, so the length of the run has to be fixed before the first step is taken. A run stopped early has not finished its decay, and the model it leaves behind is worse than one trained to that shorter length deliberately. The budget is a commitment, not an estimate.

WSD holds the peak rate flat for an unspecified duration and decays only over the final few percent. Because the flat phase has no predetermined end, the weights can be saved at any point and decayed separately from there. One run yields finished models at many budgets, and then carries on.

A 2026 result: the decay and weight averaging do the same job

Hold the learning rate constant for the whole run and keep an exponential moving average of the weights — row 10's mechanism applied to w rather than to g — and the averaged weights match the cosine schedule's loss at every point along the run.

row 10:   m ← βm + (1 − β)g          averaging the gradient    → a cleaner direction
here:     w̄ ← βw̄ + (1 − β)w          averaging the weights     → a cleaner position

Both the decay phase and the weight average are doing one thing: suppressing the noise that a large step leaves in the weights. Decaying η shrinks the noise as it is produced; averaging removes it after the fact. Row 13's result is the third face of the same coin, since decoupled weight decay already makes the finished weights an average over 1/(ηλ) steps.

Practically this is attractive for the same reason WSD is: a constant rate has no fixed horizon, so an averaged copy of the weights is a finished model at any moment, with no decay phase to run and no budget declared up front.

The three schedules on one axis

Drag the stopping point and the planned length independently. That separation is the whole argument: cosine's shape depends on a number chosen before the run starts, and the other two do not.

cosine     η(t) = peak · ½(1 + cos(πt/T))                       needs T in advance
WSD        η(t) = peak                     for t < 0.9T
                  peak · (T − t)/(0.1T)     for t ≥ 0.9T          T only sets when to start decaying
constant   η(t) = peak                                              no T at all

weight EMA    w̄ ← βw̄ + (1 − β)w   with β = 1 − 1/N over a window of N steps
              residual noise = √(1/(2N − 1))    assuming step-to-step noise is independent

How to read it

• Move the stop marker left of T. The cosine card reports what fraction of the peak rate it was still at when the run ended. Anything above zero means the model was caught mid-decay, and a run planned for that many steps would have finished lower.
• The dashed line is what WSD does from the checkpoint: decay over 0.1T more steps and finish a model whose real budget is the stop step plus that decay. Nothing before the checkpoint had to know it was coming.
• The constant line never moves, and its card is the 2026 result — the averaged weights are a finished model at every step, with the noise reduction that averaging buys reported for the same window length WSD spends decaying.
• Drag T instead. Cosine's entire curve redraws, so the learning rate at step 45,000 changes even though nothing about the first 45,000 steps of training changed. That is the structural constraint made visible.

What the drop looks like

Published WSD curves all show the same shape: several runs share one stable phase, sit above the cosine curve while it lasts, and each one falls sharply the moment its decay begins. The drop is not a gradual improvement arriving on schedule, it is a discontinuity in the curve.

The risks of WSD, specifically

The gain is real and reproduced widely. The costs are all of the same kind: WSD removes your ability to tell how the run is doing while it is doing it.

Two of these are worth separating from the rest. The instability risk is genuinely technical: the stable phase is the longest continuous exposure to the largest learning rate the run will ever use, and row 15's ratio widget shows what a high rate does to the update-to-weight ratio. Everything else is an information problem — you are choosing to defer all your evidence to the last ten percent in exchange for a lower final loss.

The anneal, and where the learning actually lands

Plot the learning rate and the loss on the same run. The loss model here is the standard one: progress accumulates with the learning rate spent so far, and the current learning rate adds a penalty on top, because a large step cannot settle into a minimum it keeps jumping across.

loss(t) = 1.0 + 4.0·(1 + P(t)/p₀)⁻⁰‧⁹ + 0.45·η(t)/ηᴸᵉᴭᴰ

P(t) = Σ η/ηᴸᵉᴭᴰ   the learning rate spent up to step t

A larger batch averages more samples into each gradient, so the estimate is closer to the true full-batch gradient and less noisy. A less noisy gradient can support a larger step. For SGD the rule is linear; for Adam it is the square root, because the update is already normalised by √v:

SGD:    η × k          when the batch grows by k
Adam:   η × √k

Why a larger batch permits a larger step

A gradient computed from a batch is an estimate. Averaging N independent samples carries 1/√N of the noise of a single one, so measured against a batch of 8 the noise is √(8/N):

A more accurate gradient can be trusted over a longer step. For gradient descent the relationship is linear, and four times the batch permits four times the rate. For Adam it is the square root, and four times the batch permits twice the rate, because Adam has already divided out the magnitude of the gradient and only its consistency remains.

Raise the batch and the amber arrow settles onto the violet one. Going from 8 to 32 pulls the mean a long way onto the truth; going from 256 to 1024 moves it very little, because the distance falls as one over the square root of the batch. Past a threshold called the critical batch size the extra samples buy almost nothing and the extra compute is spent for no return. Where that threshold sits is a property of the model and the data, and it has to be measured.

Two mechanisms, one number

Two separate things raise the global batch, and the learning rate distinguishes neither of them. Gradient accumulation runs several micro-batches one after another and steps once at the end. Data parallelism runs those micro-batches at the same time on several GPUs and averages their gradients before the step. To the optimizer these are the same change.

global batch = micro-batch × GPUs × accumulation steps

7 × 8 × 1 = 56          a micro-batch of 7 on each of 8 GPUs, no accumulation

Worked through, one change at a time

Start from a run that is already tuned. Batch 56, learning rate 0.001, and that 0.001 is known to be right for that batch.

baseline:   micro-batch 7  ×  8 GPUs  ×  1 accumulation step  =  56
            η = 0.001

Change one: accumulation from 1 to 4. Four micro-batches are run one after another and their gradients added up, and only then is a single optimizer step taken. Nothing about the hardware changed; the model simply waits four times as long before moving.

new global batch = 7 × 8 × 4 = 224
ratio to baseline = 224 / 56 = 4
η multiplier under Adam = √4 = 2
new η = 0.001 × 2 = 0.002

Change two: GPUs from 8 to 32. Four times as many machines each run a micro-batch at the same time, and their gradients are averaged before the step.

new global batch = 7 × 32 × 4 = 896
ratio to the previous batch = 896 / 224 = 4
η multiplier = √4 = 2
new η = 0.002 × 2 = 0.004

Both at once, computed from the baseline directly. The two multipliers compose, and the order they were applied in does not matter:

896 / 56 = 16          √16 = 4          η = 0.001 × 4 = 0.004   ✅

  same answer as 2 × 2, because √(4×4) = √4 × √4

Why square root for Adam and linear for gradient descent

plain gradient descent:   step = η·g              the step carries the gradient's magnitude
Adam:                     step ≈ η·(consistency)   the magnitude was divided out by √v

Under gradient descent, averaging N samples shrinks the gradient's noise and the useful part survives, so N times the batch supports N times the rate. Under Adam the magnitude is already gone; all that a bigger batch improves is how consistent the direction is, and consistency improves as √N. Applying the linear rule to an Adam run is a real mistake with a specific size to it:

What the bigger batch costs

A fixed token budget buys fewer optimizer steps as the batch grows, and √ scaling does not make that back. At a sequence length of 1024 and a billion tokens:

Sixteen times the batch, four times the rate, and a quarter of the total distance travelled. The large-batch run is not simply a faster version of the small-batch one; it takes fewer, better-aimed steps and relies on those steps being good enough. This is the same diminishing return the widget above shows geometrically, and it is why the critical batch size exists rather than being a detail: below it the extra samples buy accuracy worth having, and above it they buy a cleaner estimate of a direction that was already clean enough.

This is the step most often skipped. A team decides mid-run to accumulate gradients four times to fit a longer sequence, the effective batch quadruples, and η is left where it was. Nothing crashes; the run is simply half as effective as it should be, and nothing in the loss curve says why.

Hyperparameters are found on a small model and then a large one is trained. The problem is that the best η normally moves as the model widens, so a sweep at width 256 says little about width 4096, and a sweep at 4096 costs too much to run.

parameters per layer at width d ≈ d²(3 + 4 + 4) = 11d²

d = 256   →   11 × 65,536    =   720,896 per layer
d = 512   →   11 × 262,144   = 2,883,584 per layer      ← 4× the parameters for 2× the width

Maximal update parameterization sets the initialisation scales and per-layer learning-rate factors so that the loss-versus-η curve keeps its minimum in the same place as width grows. Sweep η at 256, 512 and 1024, confirm the three minima land together, and use that η at 4096. Without it the three curves peak in different places and there is nothing to extrapolate from.

What came after Adam

Muon and its relatives work at the matrix level rather than the scalar level, using low-rank structure in the weight updates. A 120×120 update expressed as a 120×10 times a 10×120 product costs 2,400 numbers instead of 14,400, and the compute follows. Reported gains over AdamW are real but modest, and the practical constraint is that an optimizer cannot be swapped mid-run: m and v are the accumulated history of the training so far, and discarding them discards that history.

Every optimizer that follows is an answer to one line of this table. Adam is not a cleverer descent direction — it is still −∇L. It is a per-parameter answer to the question this sheet ends on: given that these parameters have nothing in common, how does each one get its own step size without anyone hand-tuning a billion of them?