2
$\begingroup$

How does Lasso regression help with feature selection of model by making the coefficient shrink to zero?

I could see few below with below diagram. Can any please explain in simple terms how to correlate below diagram with:

  1. How Lasso shrinks the coefficient to zero
  2. How Ridge dose not shrink the coefficient to zero

enter image description here

$\endgroup$
1

3 Answers 3

2
$\begingroup$

These diagrams show the "constrained" version of lasso/ridge, in which you minimize the pure loss function subject to a constraint $\|\beta\|_1\leq t$ or $\|\beta\|_2\leq t$. (Another common version adds a penalty to the loss, and these are equivalent.)

The bluish solid shapes are the set of points with $\|\beta\|\leq t$, on the left with L1 norm and on the right with L2 norm. $\hat{\beta}$ represents the unpenalized optimum value of $\beta$, and the ovals around it are the level curves of the loss function.

We know that the optimum constrained solution will occur at a point of tangency of the level curves of the loss function and the constraint boundary, specifically with the "smallest" level curve that touches the constraint region. Then the point of the picture is that, with the pointy L1 constraint, you are much more likely for that point of tangency to occur at one of the corners, which correspond to one (or more, if we try to generalize to higher dimensions) of the coordinates in $\beta$ being zero. Compare to the L2 boundary, where the tangency being exactly along one of the axes is much less likely.

$\endgroup$
2
  • $\begingroup$ "Then the point of the picture is that, with the pointy L1 constraint, you are much more likely for that point of tangency to occur at one of the corners". But why this is the case? I mean the ellipses won't be always like that presented in the diagrams. Can you elabarotate why with the L1 constraint we are more likely to touch the corners? $\endgroup$
    – ado sar
    Commented Sep 23, 2023 at 17:37
  • $\begingroup$ @adosar Think about growing or shrinking the blue region (changing the regularization strength): with L2, there are just brief moments when the solution is on an axis; with L1, there are swaths of time. $\endgroup$
    – Ben Reiniger
    Commented Sep 26, 2023 at 13:25
1
$\begingroup$

This StatQuest video does a fantastic job of explaining in simple terms why this is the case.

$\endgroup$
0
$\begingroup$

The squared terms in Ridge Regression means that when the [Beta] coefficients shrink to small values << 1 they become essentially/effectively zero - and thus do not need to be shrunk further.

However for Lasso and its direct/linear regularization coefficients the shrinkage will need to actually bring them all the way to zero to have the same effect on the penalty sums.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.