0
$\begingroup$

I performed a multiple linear regression on 64 variables with 3 different models:

  1. Performed Multiple Linear Regression on all 64 variables
  2. Perform Feature Selection with Random Forest and then perform multiple linear regression on selected features
  3. Performed Stepwise Linear Regression

I achieved the same adjusted R squared value for all 3 models but different significant variables. How should I make sense of this? Which model should I go with?

Will appreciate any advice! Thank you!

$\endgroup$
1
  • $\begingroup$ What is the purpose of your regression? Prediction? Extrapolation? Inference? $\endgroup$
    – Spacedman
    Jun 10, 2019 at 10:43

2 Answers 2

0
$\begingroup$

It seems that removing features was not really helpful in bringing up the model fit. Differences in significance of features may be due to exclusion.

One thing you should try is regulation by lasso/ridge regression. In R, this can be easily implemented by the glmnet package. Here is a tutorial. This is the best feature selection method imo since there is a mathematical rational behind (see Introduction to Statistical learning for more background).

Lasso can shrink features to zero (viz drop them). Ridge cannot shrink features to zero. Just give it a try.

Hint: In linear regression with continuous features you can also add polynomials to your regression to increase fit using the poly function. You could also see if regression splines help you to deal with hidden non-linearity. The gam packages is a really good start. Here are the docs.

The book Introduction to Statistical Learning covers these topics in a very good way and comes with useful R examples. Here is the code.

$\endgroup$
0
$\begingroup$

64 variables is a lot for linear regression and I'd worry deeply about collinearity, interdependen variables, etc.

While a good basic assumption would be to go with the model with the fewest variables (adjusted R² being equal) I would urge you to go deeper here.

Have you performed factor analysis or PCA on the predictor variables before, would a simplified model using components or factors perform stronger and be more interpretable?

Regression really isn`t a good model to use if you just want to throw the bag at it. Depending on the motivation behind your problem (as @Spacedman pointed out) I would try more alternative models as well.

E.g. why use RF only for feature selection, why not for the whole regression? If you aim for prediction and predictive quality R² wouldn't be your main metric to look at anyway and you could try more algorithms like XGboost as well.

$\endgroup$
4
  • $\begingroup$ Why wouldn't $R^2$ be the main metric of interest? $\endgroup$
    – Dave
    Dec 2, 2021 at 22:14
  • $\begingroup$ @Dave as I said it depends on the use case. But for prediction r^2 isn't helpful because it shows the fit of the function to the sample data and not it's ability to predict unknown data. For this applying metrics like RMSE/Accuracy, etc. to predictiona the model obtained for a validation set is more helpful. E.g. if I have a biased sample my regression might have a deterministic correlation with the y and therefore r^2 of 1 but be absutely overfitted and useless for prediction. $\endgroup$
    – Fnguyen
    Dec 2, 2021 at 22:37
  • $\begingroup$ $RMSE$ and $R^2$ are two ways of telling the same story. $$R^2 = 1 - \dfrac{RMSE^2}{\sum(y_i - \bar y)^2}$$ So why wouldn't $R^2$ be of interest? $\endgroup$
    – Dave
    Dec 2, 2021 at 22:48
  • $\begingroup$ @Dave as said the r^2 you get by calculating a regression model tells you something different then the RMSE/MAPE/etc. of your prediction of unknown data. My comment was pointing out that comparing the r^2 of a model for it's sample data isn't a good way to judge a predictive model. Yes you could also calculate r^2 between predictions and validation set but in practice I've yet to see this being done over applying metrics like RMSE/MAPE/etc from the different validation libraries in python or r. $\endgroup$
    – Fnguyen
    Dec 2, 2021 at 22:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.