I have a model with 7 features, I'm trying to figure out if I can improve the performance of this model by adding additional features. So I'm relying on the RMSE to measure the accuracy of my predictions. from 7 features I get to 25 features and with each time I add a new feature, the RMSE slightly gradually get better (smaller). I find it hard to believe that all of these features improved the performance of my model as some of them have very low correlation with the target.

My question is I guess: Can I rely on the RMSE in this case to select/add features to my model?

  • $\begingroup$ Is the RMSE lower in-sample or out-of-sample? $\endgroup$
    – Dave
    Commented Jul 6, 2021 at 18:29
  • $\begingroup$ @Dave, the RMSE for the training sample is around 11 while in the testing set is around 14. $\endgroup$ Commented Jul 7, 2021 at 7:10
  • $\begingroup$ @Dave, does this imply that my model is over fitting? I used kfold cross validation to avoid over fitting $\endgroup$ Commented Jul 7, 2021 at 10:49
  • 1
    $\begingroup$ How does your out-of-sample RMSE behave as you add more and more features? $\endgroup$
    – Dave
    Commented Jul 7, 2021 at 10:50
  • $\begingroup$ @Dave, out of sample RMSE for 7 features: 14.604 ; for 9: 14.595 ; for 10: 14.584 ; for 11: 14.567; so it's improving gradually with more features. $\endgroup$ Commented Jul 7, 2021 at 10:57

1 Answer 1


The number of features can be used to handle two situations:

High bias (the common one): Adding features is one way to approach models with high bias because additional features can increase the predictive power of your data. This is commonly done as part of feature engineering.

High variance (the uncommon one): However, a large number of features can also cause overfitting and in cases with high variance reducing the number of features may reduce the extend your models overfits. This is not very common because usually eliminating features is done to get rid of irrelevant or redundant features but usually the goal is not to reduce variance (rather other techniques, such as reducing model complexity through regularization, are applied).

Therefore, it is not totally unexpected that your model performance improves with more features but it is important to check if this performance increase generalizes too (e.g. by applying cross validation) and you do not just overfit.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.