1
$\begingroup$

Is it possible to understand why Lasso models eliminated specific coefficients?. During the modelling, many of the highly correlated features in data is being eliminated by Lasso regression. Is it possible why precisely these features are being eliminated from the model? (Is it the presence of any other features/multicollinearity etc.? I want to explain the lasso model behaviour. Your help is highly appreciated.

$\endgroup$

1 Answer 1

1
$\begingroup$

Have a look at "Introduction to Statistical Learning" (Chapter 6.2.2). The Lasso adds an aditional penalty term to the original OLS penalty. In addition to the residual sum of squares (RSS, which is minimized in OLS), there is an additional term, the shrinkage penalty. Basically, coefficients which do not contribute much to a "good fit" are shrunken. Actually there is a set of $\beta_j$ for each $\lambda$ (the tuning parameter).

Not knowing your data, I can only guess that many of your highly correlated features "do not contribute much" to a "good fit" in your model. This may be a consequence from multicollinearity. However, the Lasso does not per se "shrink" features with high correlation. It rather looks for a subset of coefficients $\beta$ which produce a "good fit".

You may check whether multicollinearity is a severe issue in your data, e.g. by looking at the Variance Inflation Factor (VIF) and remove possible features with (very) high correlation.

enter image description here

$\endgroup$
2
  • $\begingroup$ Thanks, @Peter. How come highly correlated features "do not contribute much" to a "good fit" in your model?. $\endgroup$
    – NAS_2339
    Feb 16 at 16:25
  • $\begingroup$ Hard to tell not knowing the data. May be that they do not predict well in the first place. May also be that the model is "spoofed" by correlation. That's why I suggested to use VIF and remove "redundant" features so to get rid of multicollinearity. $\endgroup$
    – Peter
    Feb 16 at 16:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.