1
$\begingroup$

I am trying to fit a LR model with an obvious objective to find a best fit. model which can achieve lowest RSS.

I have many independent variable so i have decided to yous Backward selection (We start with all variables in the model, and remove the variable with the largest p-value—that is, the variable that is the least statistically significant. The new (p − 1)-variable model is fit, and the variable with the largest p-value is removed. This procedure continues until a stopping rule is reached.) to fit the model.

This is a preview of my model fit Model summary with all variables

after fitting my model i started eliminating all the variables with with high p values.

model summary after removing insignificant variable

The Adjusted R squared and RSE are almost the same in both cases indicating little to no improvement.

How should i approach it further??

$\endgroup$
  • $\begingroup$ Why are you expecting large changes in R^2 or RSE? $\endgroup$ – Ben Reiniger Jan 8 at 3:35
  • $\begingroup$ If the model fit is better my predictions will be better. Thoughts?? $\endgroup$ – Ayush Ranjan Jan 8 at 13:40
  • $\begingroup$ That depends on how you interpret "better fit", see bias-variance tradeoff. But I meant: why should removing features improve the fit (on training data)? $\endgroup$ – Ben Reiniger Jan 8 at 13:44
  • $\begingroup$ Having all the features in the model which does not help to explain the variance is also not helping the model in any way. $\endgroup$ – Ayush Ranjan Jan 8 at 21:46
  • $\begingroup$ Right: removing low-importance variables will probably hurt the fit slightly on the training data (exactly what you're seeing), but possibly/hopefully improve the predictive performance. It's not clear to me what you're looking for here. $\endgroup$ – Ben Reiniger Jan 8 at 22:28
1
$\begingroup$

Solution: Residual Plots

What is R2

The definition of R-squared is fairly straight-forward; it is the percentage of the response variable variation that is explained by a linear model.

R2 = Explained variation / Total variation

R2 is always between 0 and 100%:

  • 0% indicates that the model explains none of the variability of the response data around its mean.
  • 100% indicates that the model explains all the variability of the response data around its mean.

Limitations

R2 value has limitations. You cannot use R2 to determine whether the coefficient estimates and predictions are biased, which is why you must assess the residual plots.

R2 does not indicate if a regression model provides an adequate fit to your data. A good model can have a low R2 value. On the other hand, a biased model can have a high R2 value!

Interpreting Residual Plots

A residual is a difference between the observed y-value (from scatter plot) and the predicted y-value (from regression equation line).

A residual plot is a graph that shows the residuals on the vertical axis and the independent variable on the horizontal axis. If the points in a residual plot are randomly dispersed around the horizontal axis, a linear regression model is appropriate for the data; otherwise, a non-linear model is more appropriate.

Good Fit

enter image description here


Bad Fit

enter image description here


An unbiased model has residuals that are randomly scattered around zero. Non-random residual patterns indicate a bad fit despite a high R2.

High R2 and Bad Fit Example

Refer fitted line plot and residual plot below. It displays the relationship between semiconductor electron mobility and the natural log of the density for real experimental data.

enter image description here

Here R-squared is 98.5%. However, look closer to see how the regression line systematically over and under-predicts the data (bias) at different points along the curve. You can also see patterns in the Residuals versus Fits plot, rather than the randomness that you want to see. This indicates a bad fit. Always check residual plots!

Source and References

  1. Stattrek.com. (2010). Residual Analysis in Regression. [online] Available at: https://stattrek.com/regression/residual-analysis.aspx.
  2. Roberts, D. (2019). Residuals - MathBitsNotebook(A1 - CCSS Math). [online] Mathbitsnotebook.com. Available at: https://mathbitsnotebook.com/Algebra1/StatisticsReg/ST2Residuals.html.
  3. Coursera. (2018). Model Evaluation using Visualization - Model Development | Coursera. [online] Available at: https://www.coursera.org/learn/data-analysis-with-python/lecture/istf4/model-evaluation-using-visualization [Accessed 9 Jan. 2020].
  4. Minitab Blog Editor (2013). Regression Analysis: How Do I Interpret R-squared and Assess the Goodness-of-Fit? [online] Minitab.com. Available at: https://blog.minitab.com/blog/adventures-in-statistics-2/regression-analysis-how-do-i-interpret-r-squared-and-assess-the-goodness-of-fit.
  5. Frost, J. (2019). Jim Frost. [online] Statistics By Jim. Available at: https://statisticsbyjim.com/regression/interpret-r-squared-regression/.

‌ ‌ ‌

|improve this answer|||||
$\endgroup$
0
$\begingroup$

You should do step selection by step selection this work on AIC and tries to improve it continuously. In step selection you must try both(which dies combination of backward and forward selection both).

Once you get the final model with improved performance.. You can attempt predictions. And then further improve accuracy and f1 scores by changing the cutoff value.

If you are looking to improve the performance further, go for XGBOOST.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ He wants to do a linear regression, not expensive XGboost $\endgroup$ – Peter Jan 9 at 8:33
0
$\begingroup$

I think you look for "Forward and Backward Stepwise Selection" and related model selection techniques. Have a look at "Introduction to Statistical Learning" (pages 244-251).

The book comes with R labs, so that it should be very easy for you to apply whatever you find suitable for your task.

Especially ALSO have a look at Lasso/Ridge: The technique is able to "shrink" features of little importance, so that is possible to increase fit in many cases.

Also have a look at GAM. They are linear models, but they are able to fit so highly non-linear data. However, given this method, interpretation of coefficients is a little daunting.

So if you are interested in interpretable coefficients, you could opt for adding polynomials (i.e. squared terms) to your model or take logs (in case all observations are positive).

|improve this answer|||||
$\endgroup$
  • 1
    $\begingroup$ Thank you!! that was helpful $\endgroup$ – Ayush Ranjan Jan 9 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.