Can ridge regression be used for feature selection?

Question

I'm trying to figure out whether using Ridge Regression for regularization can be used to cause a more sparse hypothesis however to me it seems like ridge will never actually bring any coefficients to zero, only really close to it.

So can ridge regression cause any coefficients to become zero? Can the number of zeros in a weights vector change from zero to something else? Or maybe to put it simply, are the number of zero coefficients using ridge regression monotonically increasing or possibly decreasing?

Thanks.

Answer 1

Unlike lasso, ridge does not have zeroing coefficients as a goal, and you shouldn't expect applying ridge penalty to have this effect. So the answer to your title question is "no."

However, in your question body, you ask whether it is possible for the ridge penalty to produce a zero coefficient that was nonzero in an unpenalized solution. The answer here is "yes," but only as an incredible coincidence (which explains why the answer to the title question is no).

See the image in this answer (also floating around in plenty of other places). If the (unpenalized) error's contours happen to meet the constraint circle tangentially on one of the axes, that variable's coefficient will become zero. This would be an incredible coincidence, but it is theoretically possible. (Regularization can even switch the sign on the coefficient!)

I've put together a toy example to show this. GitHub/Colab notebook.
(In sklearn, we're used to thinking about regularized regression in terms of the Lagrangian form; for these kinds of diagrams, it's perhaps better to think in the constrained optimization form. See the connection e.g. here)
Let $X=\begin{pmatrix}1 & 1 \\ \sqrt{5} & -\sqrt{5} \end{pmatrix}$, $y=\begin{pmatrix}3 \\ -\sqrt{5}\end{pmatrix}$. There is an exact solution, $y=X\begin{pmatrix}1\\2\end{pmatrix}$, so the unpenalized loss contours are (not axis-aligned) ellipses centered at $(1,2)$. When the L2 penalty coefficient $\lambda$ is 5, the solution is $(0,0.5)$. When $0<\lambda<5$, the solution has first weight positive, and when $\lambda>5$ the first weight is negative(! Taking this coefficient slightly negative allows us to decrease the second coefficient even smaller, lowering the overall penalty).

Answer 2

Regularization of Ridge causes its weight to become very close to zero, but not zero. In contrast lasso can make weights equal to zero because of the types of regularization they use.

I would recommend to use lasso if you have lots of features, and you think just few of them are important. Otherwise, even though your model becomes simpler you will have a poor accuracy.

There is ElasticNet which combines both, but it is more expensive since it uses two regularization.

I didn't really understand what you ask by

Can the number of zeros in a weights vector change from zero to something else?

If you determined number of parameters beforehand their weights are unknown yet, not zero. That's what you are trying to find.

One last thing, for feature selection there are other methods. These(ridge, lasso) are just linear models for regression. If you want to identify which features work best etc. I suggest you to do a research on methods for this.

Answer 3

As mentioned by others, only Lasso can shrink parameters to exactly zero (while Ridge or Elastic Net will not, see Ch. 6.2.2 in Introduction to Statistical Learning). Lasso has some advantages, i.e. it can be used to deal with high dimensional data. For feature selection, some use a "double Lasso" approach.

In case you only want to do feature selection (or best subset selection), there are also other possibilities beyond Lasso, namely forward or backward stepwise selection.

The book "Introduction to Statistical Learning" (Chapter 6: Linear Model Selection and Regularization) gives a really instructive overview of model selection techniques.

The code examples are online for R and Python.