I have a model with 7 features, I'm trying to figure out if I can improve the performance of this model by adding additional features. So I'm relying on the RMSE to measure the accuracy of my predictions. from 7 features I get to 25 features and with each time I add a new feature, the RMSE slightly gradually get better (smaller). I find it hard to believe that all of these features improved the performance of my model as some of them have very low correlation with the target.
My question is I guess: Can I rely on the RMSE in this case to select/add features to my model?