Regression performance with Feature Selection

Question

I would like to ask you a theoretical question. In my project I am trying to get a better performance from my regression model by feature selection methods, especially with CatBoost feature importances.

I would like to ask: 1- I know the term "Garbage in Garbage out", so more features do not always mean better performance; moreover it decreases the performance. But can we get a better evaluation score like MSE, RMSE by eliminating less important features from the model? In my project It wasn't the case; MSE increased gradually while I am removing features step by step.

Should I expect a better model and higher predictive performance by eliminating unnecessary features? Or when should I expect that?

2- R2 is another metric, but I think it won't increase by eliminating a feature. In my opinion an extra feature may increase R2 value or do not effect at all. Am I right or an unnecessary feature may decrease R2?

Answer 1

For #1, Catboost includes regularization. Having a lot of features can be handled through this regularization, no need to remove features solely for improving accuracy of the model.
With proper regularization techniques, many models are able to handle having many features in the input dataset. If the model does not have regularization, then feature selection becomes more important.

For #2, you are correct. Adding more features will continually increase Rsq. You should consider using adjusted Rsq to find the point where adding features has minimal effect. It takes into consideration the number of features in the model. The more features needed to get a certain R-squared values, the lower the Adjusted R-Squared value:

Adj Rsq = 1-(1-Rsq)*(n-1)/(n-p-1)

p=number of features
n=number of data points

Key point: "the adjusted R2 increases only when the increase in R2 (due to the inclusion of a new explanatory variable) is more than one would expect to see by chance".
Reference: https://en.wikipedia.org/wiki/Coefficient_of_determination#Adjusted_R2

Answer 2

Generally speaking, some ML algorithms include regularization (such as Ridge/Lasso Regression, Random Forest, CatBoost,...) and others don't (k-NN, Gaussian Naive Bayesian,...). Using one that includes regularization may not change the performance when removing less important features.

This applies to your first question. If you use Lasso regularization for example, it will push the weights of "less important" features to zero. So removing them should not change the performance. Using Ridge regularization is a bit different.

So when you say:

MSE increased gradually while I am removing features step by step

There could be multiple reasons for that:

• the features you removed contained information and were not "less important" for the regression (see *Note/warning)
• even the less important features contain information
• you did not retrain your model after removing the "less important" features

Regarding your second question, $R^2$ is related to $MSE$ (it is well explains in this post), so if the $MSE$ increases, $R^2$ should decrease as well.

*Note/warning: a feature is often said to be "less important" using one algorithm (such as CatBoost which often uses Gini or entropy). It does not always mean this feature will be "less important" when using it with another algorithm (such as regression).