# How many features do I select when doing feature selection for regression algorithms? Is R2 and RMSE good measures of success for overfitting?

Context: I'm currently crafting and comparing machine learning models to predict housing data. I have around 32000 data points, 42 features, and I'm predicting housing price. I'm comparing Random Forest Regressor, Decision Tree Regressor, and Linear Regression. I can tell there is some overfitting going on, as my initial values vs cross validated values are as follows:

RF: 10 Fold R Squared = 0.758, neg RMSE = -540.2 vs unvalidated R Squared of 0.877, RMSE of 505.6

DT: 10 Fold R Squared = 0.711, neg RMSE = -576.4 vs unvalidated R squared of 0.829 and RMSE of 595.8.

LR: 10 Fold R squared = 0.695, neg RMSE = -596.5 vs unvalidated R squared of 0.823 and RMSE of 603.7

I have already tuned the hyperparameters for RF and DT, so I was thinking about doing feature selection as a next step to cut down on some of this overfitting (especially since I know my feature importances/coefs).I want to do feature selection now with a filter method (i.e. pearsons) as I want to keep the features going into each model consistent.

Question: How would I decide on a number of features to choose using feature selection? Is it arbitrary? Or do I basically just remove all of them that don't have much correlation with the data? Is there a way to spit out an optimized set of features without doing grid search or random search?

Follow up question: Are the R2 and RMSE cross validated values good measures of success for overfitting comparison?

• can you clarify what do you mean with "initial values vs cross validated values"? On which dataset do you compute the R2? If you divide in training-set and test-set, you do cross validation on training-set, and you report the results on the test set, I do not see why you claim there is overfitting.
– A M
Jan 14, 2021 at 12:50
• I computed the R2 on the predicted results vs the test results, and then computed the mean R2 as a measure when using cross validation over the 10 folds like so :cv_r2_scores_rf = cross_val_score(rfc, X, y, cv=10,scoring='r2', n_jobs =-1) print(cv_r2_scores_rf) print("Mean 10-Fold R Squared: {}".format(np.mean(cv_r2_scores_rf))). I think it's overfitting as the measures are much lower for the cross validation results than the y test vs prediction results. Jan 15, 2021 at 3:54
• Note that $R^2$, $MSE$, and $RMSE$ are equivalent metrics: if a model outperforms another model on one, it outperforms that model on the other two. // $R^2$ isn’t an invalid metric (since it is equivalent to $MSE$), but it loses its “percent of the variability explained” interpretation in all but the simplest settings (nonlinear regression and even regularized linear regression break this).
– Dave
Jan 15, 2021 at 12:40