The L2 regularization lead to minimize the values in the vector parameter. The L1 regularization lead to setting some coefficients to 0 in the vector parameter.

More generally, I've seen that non-differentiable regularization function lead to setting coefficients to 0 in the parameter vector. Why is that the case?


Look at the penalty terms in linear Ridge and Lasso regression:

Ridge (L2):

enter image description here

Lasso (L1):

enter image description here

Note the absolute value (L1 norm) in the Lasso penalty compared to the squared value (L2 norm) in the Ridge penalty.

In Introduction to Statistical Learning (Ch. 6.2.2) it reads: "As with ridge regression, the lasso shrinks the coefficient estimates towards zero. However, in the case of the lasso, the L1 penalty has the effect of forcing some of the coefficient estimates to be exactly equal to zero when the tuning parameter λ is sufficiently large. Hence, much like best subset selection, the lasso performs variable selection."


  • $\begingroup$ This, I know already, I'm asking why that is the case, more like a mathematical intuition or proof. $\endgroup$
    – asnart
    Jul 1 '19 at 9:09
  • 1
    $\begingroup$ I thought you can directly see it from the math: Have a look at this page (Section: Comparing regularization techniques — Intuition). It's all about the norms (L1 vs. L2) blog.alexlenail.me/… $\endgroup$
    – Peter
    Jul 1 '19 at 11:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.