0
$\begingroup$

Recently, I started working on Ridge and Lasso regularization for Linear and Logistic Regression. My doubts are given below:

  1. Is the penalty the same (by same proportion) for all the coefficients or is it based on variable importance? If it is the latter I believe we can directly apply regularization rather than spending time in feature selection.
  2. Whether the Multi-collinearity is taken care by ridge and lasso regularization?

Thank you.

|improve this question
$\endgroup$
  • 2
    $\begingroup$ 1. The penalty is the $L_p$ norm, so large features are punished. The response variable is not taken into account. 2. LASSO is sensitive to it, so it is common to combine it with ridge regression to yield "elastic net" regression (L1+L2). This selects the entire collinear group. If you don't want that, drop the redundant ones first. $\endgroup$ – Emre May 8 '18 at 4:54
  • $\begingroup$ I think Emre answers the question. $\endgroup$ – HelloWorld May 8 '18 at 5:02
1
$\begingroup$

The penalty of both Lasso and Ridge is proportional to the magnitude of the weight. That is, the penalization added to the cost function is $\lambda ||\omega||_2$ or $\lambda ||\omega||_1$.

Wether it is more convenient to apply regularization or feature selection, Lasso already does some feature selection for you, as the estimated weights for Lasso are sparse (there will be many coefficients equal to 0).

About multi-colinearity, Ridge tends to eliminate variables that are colinear, while Lasso doesn't.

All that I have said is taken from An Introduction to Statistical Learning book, from James, Witten, Hastie and Tibshirani.

|improve this answer
$\endgroup$
  • $\begingroup$ Thanks for the reply. so if the magnitude is more then the penalty would be less and vice versa correct ? and also we need to perform normalization of the features when using Lasso and Ridge right ? $\endgroup$ – deepguy May 8 '18 at 15:08
  • $\begingroup$ Did I answer your question? $\endgroup$ – David Masip May 8 '18 at 15:09
  • $\begingroup$ Masp Yes you did. Thank you. Can you also please clarify the doubts i put in the previous question. $\endgroup$ – deepguy May 8 '18 at 15:13
  • $\begingroup$ Well, I saw it and it looks interesting, but I don't have a very good answer. All that I know is that in tree-based models colinearity is not a big problem. For this reason, I wouldn't worry. However, I do not know how to explain why. $\endgroup$ – David Masip May 8 '18 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.