I've read that there are various assumptions associated with a multiple linear regression model which you should check/validate before getting too excited about your model results.

One of these is the assumption of linearity. I get that you would plot the dependent variable against the independent variable and visually check for linearity, but is there a more scientific way to do this?

I have the two plots below. Looking at the first, I can see some linearity by removing the outliers. The second however, is much harder. I can *maybe* see something, but I'm not sure if this is my eyes playing tricks on me.

enter image description here

enter image description here

If I determine the second plot does not satisfy linearity, what do I do? Exclude the feature from the model?


I like GAM (Generalised additive model) with regression splines:

# Load data
a = Auto

# Run GAM with splines
g = gam(mpg~s(displacement,5)+s(horsepower,5),data=a)
plot(g, se=T)

enter image description here

The result (plot) shows you that displacement can be well approximated by a linear function for lower values (< 250 or so). However, there is a "kink" at about 250, so that overall, a linear approximation would not be very good here.

See ISL, Chapter 7 for more details. There are also Labs for Python and R where you can see code details.

Also see this example in R with simulated data for more details.

Alternatively, look at a Q–Q plot after regression, e.g. in R:

l = lm(mpg~s(displacement,5)+s(horsepower,5),data=a)
| improve this answer | |
  • $\begingroup$ I cannot understand your plots without data description and meaning of different axis. $\endgroup$ – Subhash C. Davar Jul 1 at 9:41
  • $\begingroup$ @SubhashC.Davar: I posted the complete code (including data) as R snipped, so you can have a look at it. Also see ISL for details $\endgroup$ – Peter Jul 1 at 15:20

One common metric to determine if 2 columns have a linear relationship is R-Squared. You can use a function like this to calculate the value.

rsq <- function(x, y) summary(lm(y~x))$r.squared rsq(obs, mod)

The closer the value is to 1, the more linear the relationship is.

a similar metric to use for measuring the correlation between 2 variables (linear or otherwise) would be Pearson correlation R

cor_p <- function (x, y) cor(x, y) ^ 2

The closer the absolute value is to 1 (can also be negative), the stronger the relationship is. This can be useful in many situations.

Other metrics to consider would be MSE (mean squared error) or RMSE (root mean squared error)

If the metric value is low, you can also look at transforming one of the columns and see if the transformed column is more linearly related than the original column. Some common transforms are log(), sqrt(), exp(), etc.

Also, some models are typically fine with the data as is, no need to use a transform. One example of this is any Random Forest or Decision Tree model. In any case, excluding data just because it does not have a linear relationship is usually not the best solution as you may be removing some of the variance. Some of the valid reasons to remove a feature would be low variance or low correlation to the response, sparseness/missing, etc. The model can choose to ignore the data if it doesn't help improve the results.

| improve this answer | |
  • $\begingroup$ cor(x, y) ^ 2 I think it is representing "population correlation " . Or does it mean a different concept. There are several variants of the term - correlation. You may like to go through my profile at stats.stackexchange.com $\endgroup$ – Subhash C. Davar Jul 1 at 14:26
  • $\begingroup$ @Subhash, You are correct, thanks for pointing that out. I updated my answer to be more clear about each option. $\endgroup$ – Donald S Jul 1 at 15:03
  • $\begingroup$ wish I could mark up the Answer- one time more. $\endgroup$ – Subhash C. Davar Jul 25 at 3:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.