Follow an optimisation step

THE IDEA

Read the example.

For f(x) = x²/2, the gradient is x and one gradient-descent step multiplies x by 1−η. The learning rate controls whether progress is smooth, oscillatory or unstable in this simple example.

xₖ₊₁ = xₖ − ηf′(xₖ)

Use it well

Optimising training loss and selecting a model are different tasks. Hyperparameters, regularisation and generalisation need suitable validation as well as a successful numerical update.