I have a set of 25 features. I would like to choose the best features for my model. Originally, I was looking at the correlation of features with respect to response, and only taking those which are highly correlated and run a regression model. Then, using that model I would predict the outcome based on test data, and compare to actual (metric RMSE) and this would be how I assess it.
I could then add each feature in order of decreasing correlation with response to the feature set and keep calculating above.
Is there any other way I could select features? Could I e.g. run a random forest and use feature importance report from that to also select most important features? Then run regression?
What is the best way to compare each regression model to the next? There are so many metrics: AIC, BIC, ADJ $R^2$ I am confused as to which one is most simplest way to compare... in fact MSE is not even given in the sm.OLS function (stats models in python) summary: