BIZNIQUE / CALCULUS / 55
Connect L2 regularisation with curvature
Connect L2 regularisation with curvature
Inspect how a quadratic penalty changes both shrinkage and the second derivative.
Embedded Files
THE IDEA
Read the example.
Read the example.
For ½(w−z)² + ½λw², differentiation gives (1+λ)w−z. The coefficient shrinks smoothly as λ increases, and the second derivative increases from 1 to 1+λ.
w* = z/(1+λ); curvature = 1+λ
Use it well
Use it well
L2 shrinkage usually does not set a nonzero scalar estimate exactly to zero at finite λ. The effect of regularisation depends on scaling and the objective’s coefficient convention.
TRY IT YOURSELF
A quick check.
A quick check.
With z = 3 and λ = 2, find w* and the second derivative.
Check your reasoning
Check your reasoning
w* = 1 and the second derivative is 3.
Page updated
Google Sites
Report abuse