See why an L1 penalty can create exact zeros

THE IDEA

Read the example.

For the scalar objective ½(w−z)² + λ|w|, the minimiser is soft-thresholding. In the demo z = 3. Increasing λ first shrinks the coefficient and eventually sets it exactly to zero.

w* = sign(z) max(|z|−λ, 0)

Use it well

The simple closed form assumes this scalar quadratic or an equivalent orthogonal setting. Correlated features and general losses require solving a different optimisation problem.