variables selection in regression models

Question

I develop price prediction data model using multiple linear regression, ridge, lasso and elastic net regression, initially I had 215 variables. after creating models I ran a python code to check how many variables have used in final models, this is python code which i use for the detect number of variables in ridge regression,

print("Ridge Regression Selected " + str(sum(coef_ridge != 0)) + " Variables and Neglected " +  str(sum(coef_ridge == 0)) + " Variables")

This is a out put which I got

Ridge Regression Selected 209 Variables and Neglected 6 Variables

above code ran for all other modeling methods separably. then Multiple regression 212 variables selected, Ridge 209 variables selected, Lasso 68 variables selected and Elastic net regression 77 variables selected.

my question is according to my knowledge lasso and elastic net regression already use variables selection method because of that, number of selected variables reduced up to 68 and 77 but how multiple linear regression neglect 3 variables and ridge regression reduce 6 variables? that mean multiple and ridge regression also use variable selection method in their code? please someone explain these scenario

Answer 1

Lasso stands for ´least absolute shrinkage and selection operator´. It has a penalty that is the absolute value and makes a lot of variables converge to cero. There is a ton of blogs that explain really well Lasso on the internet, have a look!

Elastic Net is a combination of Ridge and Lasso. So it will also reduce the variables a lot.

Ridge is a quadratic penalty so the convergence is mellower. There are also some posts about why this happens Why will ridge regression not shrink some coefficients to zero like lasso?

In Multiple Linear regression, there is no penalty, so coefficients have not a special reason to be zero. They will only be cero when there is no relation with the target variable. Imagine that you add a feature that is just random noise. You will end up with a cero coefficient for your linear model.

This should give you an intuition of why Lasso shrinks most of the features, Enet a little bit less, Ridge not so many and Linear Regression only some or none.