Linear Regression finding best fit

Question

I am trying to fit a LR model with an obvious objective to find a best fit. model which can achieve lowest RSS.

I have many independent variable so i have decided to yous Backward selection (We start with all variables in the model, and remove the variable with the largest p-value—that is, the variable that is the least statistically significant. The new (p − 1)-variable model is fit, and the variable with the largest p-value is removed. This procedure continues until a stopping rule is reached.) to fit the model.

This is a preview of my model fit

after fitting my model i started eliminating all the variables with with high p values.

The Adjusted R squared and RSE are almost the same in both cases indicating little to no improvement.

How should i approach it further??

Answer 1

Solution: Residual Plots

What is R²

The definition of R-squared is fairly straight-forward; it is the percentage of the response variable variation that is explained by a linear model.

R² = Explained variation / Total variation

R² is always between 0 and 100%:

• 0% indicates that the model explains none of the variability of the response data around its mean.
  
• 100% indicates that the model explains all the variability of the response data around its mean.

Limitations

R² value has limitations. You cannot use R² to determine whether the coefficient estimates and predictions are biased, which is why you must assess the residual plots.

R² does not indicate if a regression model provides an adequate fit to your data. A good model can have a low R² value. On the other hand, a biased model can have a high R² value!

Interpreting Residual Plots

A residual is a difference between the observed y-value (from scatter plot) and the predicted y-value (from regression equation line).

A residual plot is a graph that shows the residuals on the vertical axis and the independent variable on the horizontal axis. If the points in a residual plot are randomly dispersed around the horizontal axis, a linear regression model is appropriate for the data; otherwise, a non-linear model is more appropriate.

Good Fit

---

Bad Fit

---

An unbiased model has residuals that are randomly scattered around zero. Non-random residual patterns indicate a bad fit despite a high R².

High R² and Bad Fit Example

Refer fitted line plot and residual plot below. It displays the relationship between semiconductor electron mobility and the natural log of the density for real experimental data.

Here R-squared is 98.5%. However, look closer to see how the regression line systematically over and under-predicts the data (bias) at different points along the curve. You can also see patterns in the Residuals versus Fits plot, rather than the randomness that you want to see. This indicates a bad fit. Always check residual plots!

Source and References

1. Stattrek.com. (2010). Residual Analysis in Regression. [online] Available at: https://stattrek.com/regression/residual-analysis.aspx.
2. Roberts, D. (2019). Residuals - MathBitsNotebook(A1 - CCSS Math). [online] Mathbitsnotebook.com. Available at: https://mathbitsnotebook.com/Algebra1/StatisticsReg/ST2Residuals.html.
3. Coursera. (2018). Model Evaluation using Visualization - Model Development | Coursera. [online] Available at: https://www.coursera.org/learn/data-analysis-with-python/lecture/istf4/model-evaluation-using-visualization [Accessed 9 Jan. 2020].
4. Minitab Blog Editor (2013). Regression Analysis: How Do I Interpret R-squared and Assess the Goodness-of-Fit? [online] Minitab.com. Available at: https://blog.minitab.com/blog/adventures-in-statistics-2/regression-analysis-how-do-i-interpret-r-squared-and-assess-the-goodness-of-fit.
5. Frost, J. (2019). Jim Frost. [online] Statistics By Jim. Available at: https://statisticsbyjim.com/regression/interpret-r-squared-regression/.

‌ ‌ ‌

Answer 2

You should do step selection by step selection this work on AIC and tries to improve it continuously. In step selection you must try both(which dies combination of backward and forward selection both).

Once you get the final model with improved performance.. You can attempt predictions. And then further improve accuracy and f1 scores by changing the cutoff value.

If you are looking to improve the performance further, go for XGBOOST.

Answer 3

I think you look for "Forward and Backward Stepwise Selection" and related model selection techniques. Have a look at "Introduction to Statistical Learning" (pages 244-251).

The book comes with R labs, so that it should be very easy for you to apply whatever you find suitable for your task.

Especially ALSO have a look at Lasso/Ridge: The technique is able to "shrink" features of little importance, so that is possible to increase fit in many cases.

Also have a look at GAM. They are linear models, but they are able to fit so highly non-linear data. However, given this method, interpretation of coefficients is a little daunting.

So if you are interested in interpretable coefficients, you could opt for adding polynomials (i.e. squared terms) to your model or take logs (in case all observations are positive).