Selecting most important features for multilinear regression

Question

I have a set of 25 features. I would like to choose the best features for my model. Originally, I was looking at the correlation of features with respect to response, and only taking those which are highly correlated and run a regression model. Then, using that model I would predict the outcome based on test data, and compare it to actual (metric RMSE) and this would be how I assess it.

I could then add each feature in order of decreasing correlation with response to the feature set and keep calculating above.

Is there any other way I could select features? Could I e.g. run a random forest and use a feature importance report from that to also select the most important features? Then run a regression?

What is the best way to compare each regression model to the next? There are so many metrics: AIC, BIC, ADJ $R^2$ I am confused as to which one is the simplest way to compare... in fact, MSE is not even given in the sm.OLS function (stats models in python) summary:

Answer 1

Be careful choosing features based on correlation! Yes it is true that features that are correlated with the response variable may be good predictors, however if the features are correlated with each other then you are introducing multicollinearity into your model, which is bad.

If you want to avoid this you should choose features which are correlated with your response variable but not with each other. For example...

y = a + b + c

Assume: y is correlated with all three (a, b, c)

a is correlated with b

c is not correlated with a and b

You should only use one of either a or b and c to predict y.

With regard to choosing which model is best, you should use a combination of metrics. As you increase the number of features your R^2 will tend to increase regardless of model performance. AIC and BIC can be used only to compare similar models and the "best" model according to AIC/BIC will be the one with the lowest score relative to the other models.

In my opinion RMSE is the best indicator of a good model, but you should also evaluate R^2, AIC, and BIC.

If you are unsure of which features to use I suggest you try stepwise regression to evaluate many models quickly and (maybe) find the best one.

Answer 2

Consider playing around with LASSO or Ridge-regressions, as these punish features with low predictive power. These are simple and strong methods for linear purposes.

Your idea of using the feature importance from Random Forest could also be a suitable solution in cases of non-linearity.