BIZNIQUE / CALCULUS / 56
Balance fit and two forms of regularisation
Balance fit and two forms of regularisation
Treat a penalty as part of the stated objective, with an explicit scale.
Embedded Files
THE IDEA
Read the example.
Read the example.
The scalar objective combines squared fit error, λ₁|w| and ½λ₂w². L1 contributes a threshold; L2 contributes additional shrinkage. The example keeps λ₁ + λ₂ = 3 while varying their mixture. The optimum moves from 0.75 with pure L2 to zero with pure L1.
w* = soft(z, λ₁)/(1+λ₂)
Use it well
Use it well
Penalty weights are not interchangeable across differently scaled features or loss definitions. Choose them using the intended evaluation procedure and retain an unpenalised reference.
TRY IT YOURSELF
A quick check.
A quick check.
For z = 3, λ₁ = 1.5 and λ₂ = 1.5, what is the minimiser?
Check your reasoning
Check your reasoning
soft(3,1.5)/(1+1.5) = 1.5/2.5 = 0.6.
Page updated
Google Sites
Report abuse