BIZNIQUE / CALCULUS / 52
Control the size of a gradient step
Control the size of a gradient step
Distinguish the size of an update from the validity of the gradient that produced it.
Embedded Files
THE IDEA
Read the example.
Read the example.
Global norm clipping limits a gradient vector’s norm to c while preserving its direction when it is too large. The example compares a growing scalar chain with a chosen clipping threshold.
gclip = g × min(1, c / ‖g‖₂)
Use it well
Use it well
Clipping does not repair incorrect derivatives or guarantee stable learning. It changes the update and can mask a deeper problem if used without diagnostics.
TRY IT YOURSELF
A quick check.
A quick check.
For g = (3, 4) and c = 2, what is the clipped vector?
Check your reasoning
Check your reasoning
‖g‖ = 5, so multiply by 2/5 to obtain (1.2, 1.6), whose norm is 2.
Page updated
Google Sites
Report abuse