2
$\begingroup$

I'm trying to figure out whether using Ridge Regression for regularization can be used to cause a more sparse hypothesis however to me it seems like ridge will never actually bring any coefficients to zero, only really close to it.

So can ridge regression cause any coefficients to become zero? Can the number of zeros in a weights vector change from zero to something else? Or maybe to put it simply, are the number of zero coefficients using ridge regression monotonically increasing or possibly decreasing?

Thanks.

$\endgroup$
  • 2
    $\begingroup$ Unlikely. That is what Lasso and ElasticNet are for. $\endgroup$ – steam_engine Aug 15 at 15:53
  • $\begingroup$ Any idea if it can cause any coefficients to go to zero though? I mean is it even possible? $\endgroup$ – edan patt Aug 15 at 15:55
  • $\begingroup$ Well, they might. Off the top of my head - if the target variable does not correlate with one of the features, then the corresponding weight would be 0. But that has nothing to do with ridge, same would happen with OLS, that would just be the property of data. $\endgroup$ – steam_engine Aug 15 at 16:01
3
$\begingroup$

Regularization of Ridge causes its weight to become very close to zero, but not zero. In contrast lasso can make weights equal to zero because of the types of regularization they use.

I would recommend to use lasso if you have lots of features, and you think just few of them are important. Otherwise, even though your model becomes simpler you will have a poor accuracy.

There is ElasticNet which combines both, but it is more expensive since it uses two regularization.

I didn't really understand what you ask by

Can the number of zeros in a weights vector change from zero to something else?

If you determined number of parameters beforehand their weights are unknown yet, not zero. That's what you are trying to find.

One last thing, for feature selection there are other methods. These(ridge, lasso) are just linear models for regression. If you want to identify which features work best etc. I suggest you to do a research on methods for this.

$\endgroup$
  • $\begingroup$ How about this, can a feature that started out as zero become non zero using ridge regression? $\endgroup$ – edan patt Aug 17 at 13:36
  • $\begingroup$ @edanpatt, Yes, that's also possible. See the link in my answer that explains the sign of the coefficient can change; it's not hard to adjust that example to show that a zero coefficient can become nonzero (if doing so allows other coefficients to drop enough to offset the loss). In fact, it seems rather likely in ridge regression that this will happen, whereas for lasso it takes a more special situation. $\endgroup$ – Ben Reiniger Aug 17 at 18:38
2
$\begingroup$

As mentioned by others, only Lasso can shrink parameters to exactly zero (while Ridge or Elastic Net will not, see Ch. 6.2.2 in Introduction to Statistical Learning). Lasso has some advantages, i.e. it can be used to deal with high dimensional data. For feature selection, some use a "double Lasso" approach.

In case you only want to do feature selection (or best subset selection), there are also other possibilities beyond Lasso, namely forward or backward stepwise selection.

The book "Introduction to Statistical Learning" (Chapter 6: Linear Model Selection and Regularization) gives a really instructive overview of model selection techniques.

The code examples are online for R and Python.

$\endgroup$
  • $\begingroup$ ElasticNet can and will push parameters to zero, I think. $\endgroup$ – Ben Reiniger Aug 19 at 13:40
2
$\begingroup$

Unlike lasso, ridge does not have zeroing coefficients as a goal, and you shouldn't expect applying ridge penalty to have this effect. So the answer to your title question is "no."

However, in your question body, you ask whether it is possible for the ridge penalty to produce a zero coefficient that was nonzero in an unpenalized solution. The answer here is "yes," but only as an incredible coincidence (which explains why the answer to the title question is no).

See the image in this answer (also floating around in plenty of other places). If the (unpenalized) error's contours happen to meet the constraint circle tangentially on one of the axes, that variable's coefficient will become zero. This would be an incredible coincidence, but it is theoretically possible. (Regularization can even switch the sign on the coefficient!)

I've put together a toy example to show this. GitHub/Colab notebook.
(In sklearn, we're used to thinking about regularized regression in terms of the Lagrangian form; for these kinds of diagrams, it's perhaps better to think in the constrained optimization form. See the connection e.g. here)
Let $X=\begin{pmatrix}1 & 1 \\ \sqrt{5} & -\sqrt{5} \end{pmatrix}$, $y=\begin{pmatrix}3 \\ -\sqrt{5}\end{pmatrix}$. There is an exact solution, $y=X\begin{pmatrix}1\\2\end{pmatrix}$, so the unpenalized loss contours are (not axis-aligned) ellipses centered at $(1,2)$. When the L2 penalty coefficient $\lambda$ is 5, the solution is $(0,0.5)$. When $0<\lambda<5$, the solution has first weight positive, and when $\lambda>5$ the first weight is negative(! Taking this coefficient slightly negative allows us to decrease the second coefficient even smaller, lowering the overall penalty).

$\endgroup$
  • 1
    $\begingroup$ Ben, I'm a little confused. Can you elaborate on the ridge claim? Under ridge no coefficient will be shrunken to zero unlike $\lambda = \infty$. Just had a look at Section 6.2.2 of Introduction to Statistical Learning since you confused me. (No need to say that a coef can be zero under say OLS or so, but ridge does not shrink a coef to exactly zero. $\endgroup$ – Peter Aug 16 at 17:17
  • $\begingroup$ @Peter, thanks, I've added a toy example and some explanation. Since the zeroing of a coefficient will only happen for a particular value of the penalty, I think it's still fair to say ridge "will not set any of them exactly to zero" as ISL does. I just wanted to point out that it's only practically impossible, not theoretically impossible. $\endgroup$ – Ben Reiniger Aug 17 at 16:56
  • $\begingroup$ Great! Upvote... need to look at the Github file, thx for sharing! $\endgroup$ – Peter Aug 17 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.