BIZNIQUE / CALCULUS / 44
Compare gradient descent and momentum
Compare gradient descent and momentum
Follow two update rules on the same small objective before drawing broader conclusions.
Embedded Files
THE IDEA
Read the example.
Read the example.
Ordinary gradient descent uses the current gradient directly. Momentum accumulates a decaying history of gradients, which can change both progress and overshoot. Here both methods use the same quadratic, starting point and learning rate; β is 0.6.
vₖ₊₁ = βvₖ + gₖ; xₖ₊₁ = xₖ − ηvₖ₊₁
Use it well
Use it well
No single toy plot establishes a universally best optimiser. The objective, parameterisation, noise, tuning and stopping rule affect the comparison.
TRY IT YOURSELF
A quick check.
A quick check.
With v₀ = 0, β = 0.6 and g₀ = 3, what is v₁?
Check your reasoning
Check your reasoning
v₁ = 0.6×0 + 3 = 3. The first update matches ordinary gradient descent at the same learning rate.
Page updated
Google Sites
Report abuse