1
$\begingroup$

Sparse methods such as LASSO contains a parameter $\lambda$ which is associated with the minimization of the $l_1$ norm. Higher the value of $\lambda$ ($>0$) it means that more coefficients will be shrunk to zero. What is unclear to me is that how does this method decide which coefficients to shrink to zero?

  • If $\lambda = 0.5$ then does it mean that those coefficients whose values are less than or equal to 0.5 will become zero? So in other words, whatever be the value of $\lambda$, the coefficients whose values fall within $\lambda$ will be turned off/become zero? OR is there some other meaning to the value of $\lambda$?
  • Can $\lambda$ be negative?
$\endgroup$
0
2
$\begingroup$

When we implement penalized regression models we are saying that we are going to add a penalty to the sum of the squared errors.

Recall that the sum of squared errors is the following and that we are trying to minimize this value with Least Squares Regression:

$$SSE = \sum_{i=1}^{n}(y_i-\hat{y_i})^2$$

When the model overfits or there is collinearity present, the estimates for the coefficients for our least squares model may be higher than they should be.

How do we fix this? We use regularization. What this means is that we add a penalty to the sum of squared errors thereby limiting how large the parameter estimates can get.

For Ridge Regression this looks like this:

$$SSE_{L2 norm} = \sum_{i=1}^{n}(y_i-\hat{y_i})^2 + \lambda \sum_{j=1}^{P}\beta_j^2$$

Notice what is different with this model. Here, we add a L2 regularization penalty to the end of the SSE. What this does is add the multiplication of $\lambda$ by the square of the parameter estimations as a penalty to the SSE. This limits how large the parameter estimates can get. As you increase the "shrinkage parameter" $\lambda$ the parameter estimates are shrunk more towards zero. What is important to note with Ridge Regression, is that this model shrinks the values towards zero, but not to zero.

You may also use the LASSO Regression technique as shown below:

$$SSE_{L1 norm} = \sum_{i=1}^{n}(y_i-\hat{y_i})^2 + \lambda \sum_{j=1}^{P} \lvert{\beta_j^2}\rvert$$

Notice here that the change by adding the L1 penalty is very similar. However, this difference here is that we are now penalizing the absolute value of the coefficients. This allows shrinkage to zero and can be considered a form of feature selection.

In either case both methods penalize model complexity and are parsimonious!

Two things to note:

  1. In answer to your question, no, $\lambda$ cannot be negative. Why? This would make no sense. $\lambda$ is multiplied by either the L2 or L1 norm to add a penalty to the SSE. If instead, you had a negative $\lambda$ you would actually be rewarding the model's complexity not penalizing it.

  2. When you have a value of $\lambda = 0$ you have no penalty and just have regular least squares regression!

$\endgroup$
3
  • $\begingroup$ Thank you for answering however the main point in my question is not present in your answer which is what does the value of $\lambda$ imply? When you say that the coefficients shrink to zero, which coefficients are they...how are they chosen/determined? If $\lambda = 0.1$ then does it mean that those coefficients whose values are less than or equal to 0.1 will become zero? $\endgroup$ – Sm1 Dec 1 '20 at 6:12
  • $\begingroup$ You can see from the models above that as you increase $\lambda$ you will increase the penalty. Lambda is arbitrarily chosen by the user (but can be optimized with cross validation). The coefficients that shrink to zero are the coefficients on your regression model. ie the coefficients on $\beta_1, \beta_2$ and so on. $\endgroup$ – Ethan Dec 1 '20 at 7:43
  • 1
    $\begingroup$ The intuition is that say you have a model $y = \beta + 2\beta_1 + 3\beta_2$ these coefficients may not give you the lowest variance. You can add biased estimators through regularization and attempt to lower your variance. These coefficients will be shrunk towards zero depending on their error. $\endgroup$ – Ethan Dec 1 '20 at 7:46
1
$\begingroup$

@Ethan is correct about the formulation of the lasso penalty, and I think it's particularly important to understand it in that form (for one thing, because that same penalty can work with other models like neural networks, tree models, generalized linear models, ...).

But, to your question:

If $\lambda=0.5$ then does it mean that those coefficients whose values are less than or equal to 0.5 will become zero? So in other words, whatever be the value of $\lambda$, the coefficients whose values fall within $\lambda$ will be turned off/become zero? OR is there some other meaning to the value of $\lambda$?

The answer is mostly yes, under some assumptions. For vanilla LASSO (OLS plus the L1-penalty), if the covariates are orthonormal, then the LASSO coefficients can be written in terms of the OLS coefficients:

$$\beta_i^{\text{LASSO}} = \begin{cases} \beta_i^{\text{OLS}} - \lambda & \text{if $\beta_i^{\text{OLS}} \geq \lambda$} \\ \beta_i^{\text{OLS}} + \lambda & \text{if $\beta_i^{\text{OLS}} \leq -\lambda$} \\ 0 & \text{if $|\beta_i^{\text{OLS}}|\leq\lambda$.} \end{cases}$$ (You'll see that written more concisely as $\operatorname{sgn}(\beta_i^{\text{OLS}}) \left(|\beta_i^{\text{OLS}}|-\lambda\right)_+$.)

See e.g.:
https://stats.stackexchange.com/q/17781/232706
http://pages.cs.wisc.edu/~jerryzhu/cs731/regression.pdf (section 3)
https://stats.stackexchange.com/q/342547/232706

$\endgroup$
1
  • $\begingroup$ Thank you very much for explaining the intuitive meaning of the parameter. $\endgroup$ – Sm1 Dec 5 '20 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.