Checking linearity for a linear regression model?

Question

I've read that there are various assumptions associated with a multiple linear regression model which you should check/validate before getting too excited about your model results.

One of these is the assumption of linearity. I get that you would plot the dependent variable against the independent variable and visually check for linearity, but is there a more scientific way to do this?

I have the two plots below. Looking at the first, I can see some linearity by removing the outliers. The second however, is much harder. I can *maybe* see something, but I'm not sure if this is my eyes playing tricks on me.

If I determine the second plot does not satisfy linearity, what do I do? Exclude the feature from the model?

Answer 1

I like GAM (Generalised additive model) with regression splines:

# Load data
library(ISLR)
a = Auto

# Run GAM with splines
library(gam)
g = gam(mpg~s(displacement,5)+s(horsepower,5),data=a)
plot(g, se=T)

The result (plot) shows you that displacement can be well approximated by a linear function for lower values (< 250 or so). However, there is a "kink" at about 250, so that overall, a linear approximation would not be very good here.

See ISL, Chapter 7 for more details. There are also Labs for Python and R where you can see code details.

Also see this example in R with simulated data for more details.

Alternatively, look at a Q–Q plot after regression, e.g. in R:

l = lm(mpg~s(displacement,5)+s(horsepower,5),data=a)
plot(l)

Answer 2

One common metric to determine if 2 columns have a linear relationship is R-Squared. You can use a function like this to calculate the value.

rsq <- function(x, y) summary(lm(y~x))$r.squared rsq(obs, mod)

The closer the value is to 1, the more linear the relationship is.

a similar metric to use for measuring the correlation between 2 variables (linear or otherwise) would be Pearson correlation R

cor_p <- function (x, y) cor(x, y) ^ 2

The closer the absolute value is to 1 (can also be negative), the stronger the relationship is. This can be useful in many situations.

Other metrics to consider would be MSE (mean squared error) or RMSE (root mean squared error)

If the metric value is low, you can also look at transforming one of the columns and see if the transformed column is more linearly related than the original column. Some common transforms are log(), sqrt(), exp(), etc.

Also, some models are typically fine with the data as is, no need to use a transform. One example of this is any Random Forest or Decision Tree model. In any case, excluding data just because it does not have a linear relationship is usually not the best solution as you may be removing some of the variance. Some of the valid reasons to remove a feature would be low variance or low correlation to the response, sparseness/missing, etc. The model can choose to ignore the data if it doesn't help improve the results.