Context: I'm currently crafting and comparing machine learning models to predict housing data. I have around 32000 data points, 42 features, and I'm predicting housing price. I'm comparing Random Forest Regressor, Decision Tree Regressor, and Linear Regression. I can tell there is some overfitting going on, as my initial values vs cross validated values are as follows:

RF: 10 Fold R Squared = 0.758, neg RMSE = -540.2 vs unvalidated R Squared of 0.877, RMSE of 505.6

DT: 10 Fold R Squared = 0.711, neg RMSE = -576.4 vs unvalidated R squared of 0.829 and RMSE of 595.8.

LR: 10 Fold R squared = 0.695, neg RMSE = -596.5 vs unvalidated R squared of 0.823 and RMSE of 603.7

I have already tuned the hyperparameters for RF and DT, so I was thinking about doing feature selection as a next step to cut down on some of this overfitting (especially since I know my feature importances/coefs).I want to do feature selection now with a filter method (i.e. pearsons) as I want to keep the features going into each model consistent.

Question: How would I decide on a number of features to choose using feature selection? Is it arbitrary? Or do I basically just remove all of them that don't have much correlation with the data? Is there a way to spit out an optimized set of features without doing grid search or random search?

Follow up question: Are the R2 and RMSE cross validated values good measures of success for overfitting comparison?

  • $\begingroup$ can you clarify what do you mean with "initial values vs cross validated values"? On which dataset do you compute the R2? If you divide in training-set and test-set, you do cross validation on training-set, and you report the results on the test set, I do not see why you claim there is overfitting. $\endgroup$ – A M Jan 14 at 12:50
  • $\begingroup$ I computed the R2 on the predicted results vs the test results, and then computed the mean R2 as a measure when using cross validation over the 10 folds like so :cv_r2_scores_rf = cross_val_score(rfc, X, y, cv=10,scoring='r2', n_jobs =-1) print(cv_r2_scores_rf) print("Mean 10-Fold R Squared: {}".format(np.mean(cv_r2_scores_rf))). I think it's overfitting as the measures are much lower for the cross validation results than the y test vs prediction results. $\endgroup$ – pythonnoob2 Jan 15 at 3:54
  • $\begingroup$ Note that $R^2$, $MSE$, and $RMSE$ are equivalent metrics: if a model outperforms another model on one, it outperforms that model on the other two. // $R^2$ isn’t an invalid metric (since it is equivalent to $MSE$), but it loses its “percent of the variability explained” interpretation in all but the simplest settings (nonlinear regression and even regularized linear regression break this). $\endgroup$ – Dave Jan 15 at 12:40

You have overfitting when your model corresponds too closely to the training data and may therefore fail to fit additional data or predict future observations reliably. Basically, when the performance on training (or validation) set are much more better than on test set. You have the opposite case: performance on test set much better than validation set.

This can happen if the two sets do not come from the same distribution (how did you divide train/validation/test set?). In this case, the data of the test set could be much easier to predict.

Another possibility is that the size of the test set is too small.

My suggestion is: shuffle your dataset; divide in 70% training and 30% test set. Do cross validation on training set. Compute the R2 on both sets.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.