I would like to ask you a theoretical question. In my project I am trying to get a better performance from my regression model by feature selection methods, especially with CatBoost feature importances.
I would like to ask: 1- I know the term "Garbage in Garbage out", so more features do not always mean better performance; moreover it decreases the performance. But can we get a better evaluation score like MSE, RMSE by eliminating less important features from the model? In my project It wasn't the case; MSE increased gradually while I am removing features step by step.
Should I expect a better model and higher predictive performance by eliminating unnecessary features? Or when should I expect that?
2- R2 is another metric, but I think it won't increase by eliminating a feature. In my opinion an extra feature may increase R2 value or do not effect at all. Am I right or an unnecessary feature may decrease R2?