BIZNIQUE / CALCULUS / 51
Understand vanishing gradients
Understand vanishing gradients
A chain of small local sensitivities can attenuate a signal dramatically.
Embedded Files
THE IDEA
Read the example.
Read the example.
The illustration multiplies the same scalar sensitivity a across T steps. If |a| is below one, its magnitude shrinks with depth. Neural-network gradients involve richer chains of Jacobians, but this toy case makes one source of attenuation visible.
Repeated scalar sensitivity = aᵀ
Use it well
Use it well
Activation choice, initialisation, residual paths and gated structures can help in suitable models. None removes the need to inspect gradient flow and train under realistic conditions.
TRY IT YOURSELF
A quick check.
A quick check.
What is 0.5¹⁰?
Check your reasoning
Check your reasoning
1/1024, approximately 0.000977. A modest local shrinkage has accumulated across ten steps.
Page updated
Google Sites
Report abuse