BIZNIQUE / CALCULUS / 47
Separate a noisy step from a better objective
Separate a noisy step from a better objective
Watch an illustrative disturbance change a gradient-descent path.
Embedded Files
THE IDEA
Read the example.
Read the example.
The demo adds a deterministic oscillating disturbance to the exact gradient of a quadratic. It illustrates why individual steps can fluctuate even when an underlying objective is simple. Real stochastic gradients depend on sampling and data.
Observed update direction = gradient + disturbance
Use it well
Use it well
Noise can alter exploration and stability, but it does not guarantee escape from a poor region or convergence to a better result. Inspect trends and the actual evaluation objective.
TRY IT YOURSELF
A quick check.
A quick check.
Can a single rise in training loss prove that a stochastic optimiser is broken?
Check your reasoning
Check your reasoning
No. Stochastic updates can fluctuate; persistent behaviour, step size and evaluation conditions need to be examined.
Page updated
Google Sites
Report abuse