13
$\begingroup$

I'm just getting started with machine learning and I have trouble understanding how overfitting can happen in a linear regression model.

Considering we use only 2 feature variables to train a model, how can a flat plane possibly be overfitted to a set of data points?

I assume linear regression uses only a line to describe the linear relationship between 2 variables and a flat plane to describe the relationship between 3 variables, I have trouble understanding (or rather imagining) how overfitting in a line or a plane can happen?

$\endgroup$
3
  • 1
    $\begingroup$ For two variables and a linear decision surface this will indeed not be much of a concern unless one or both variables are completely unrelated to the target. Underfitting is likely the bigger problem. (Just note that linear regression doesn't have to produce a linear decision surface, like polynomial (linear) regression, as shown in the other answers.) $\endgroup$
    – oW_
    Aug 27 '20 at 17:38
  • 2
    $\begingroup$ You don't even need to add polynomial features like @RobertLong did in order to badly overfit a linear model! datascience.stackexchange.com/a/79994/73930 $\endgroup$
    – Dave
    Aug 27 '20 at 20:33
  • 1
    $\begingroup$ To be clear, your model's $f\left(x, y\right)= a x + b y + c ,$ but someone told you that it was overfitting? Some additional background/context may help, as that claim would seem to be odd. At least, assuming independence between $x$ and $y ;$ if one's a function of the other or something, then such a model could be argued as being an overfit. $\endgroup$
    – Nat
    Aug 28 '20 at 11:11
22
$\begingroup$

In linear regression overfitting occurs when the model is "too complex". This usually happens when there are a large number of parameters compared to the number of observations. Such a model will not generalise well to new data. That is, it will perform well on training data, but poorly on test data.

A simple simulation can show this. Here I use R:

> set.seed(2)
> N <- 4
> X <- 1:N
> Y <- X + rnorm(N, 0, 1)
> 
> (m0 <- lm(Y ~ X)) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  -0.2393     1.8568  -0.129    0.909
X             1.0703     0.6780   1.579    0.255

Residual standard error: 1.516 on 2 degrees of freedom
Multiple R-squared:  0.5548,    Adjusted R-squared:  0.3321 
F-statistic: 2.492 on 1 and 2 DF,  p-value: 0.2552

Note that we obtain a good estimate of the true value for the coefficient of X. Note the Adjusted R-squared of 0.3321 which is an indication of the model fit.

Now we fit a quadratic model:

> (m1 <- lm(Y ~ X + I(X^2) )) %>% summary()


Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  -4.9893     2.7654  -1.804    0.322
X             5.8202     2.5228   2.307    0.260
I(X^2)       -0.9500     0.4967  -1.913    0.307

Residual standard error: 0.9934 on 1 degrees of freedom
Multiple R-squared:  0.9044,    Adjusted R-squared:  0.7133 
F-statistic: 4.731 on 2 and 1 DF,  p-value: 0.3092

Now we have a much higher Adjusted R-squared: 0.7133 which may lead us to think that the model is much better. Indeed if we plot the data and the predicted valus from both models we get :

> fun.linear <- function(x) { coef(m0)[1] + coef(m0)[2] * x  }
> fun.quadratic <- function(x) { coef(m1)[1] + coef(m1)[2] * x  + coef(m1)[3] * x^2}
> 
> ggplot(data.frame(X,Y), aes(y = Y, x = X)) + geom_point()  + stat_function(fun = fun.linear) + stat_function(fun = fun.quadratic)

enter image description here

So on the face of it, the quadratic model looks much better.

Now, if we simulate new data, but use the same model to plot the predictions, we get

> set.seed(6)
> N <- 4
> X <- 1:N
> Y <- X + rnorm(N, 0, 1)
> ggplot(data.frame(X,Y), aes(y = Y, x = X)) + geom_point()  + stat_function(fun = fun.linear) + stat_function(fun = fun.quadratic)

enter image description here

Clearly the quadratic model is not doing well, whereas the linear model is still reasonable. However, if we simulate more data with an extended range, using the original seed, so that the initial data points are the same as in the first simulation we find:

> set.seed(2)
> N <- 10
> X <- 1:N
> Y <- X + rnorm(N, 0, 1)
> ggplot(data.frame(X,Y), aes(y = Y, x = X)) + geom_point()  + stat_function(fun = fun.linear) + stat_function(fun = fun.quadratic)

enter image description here

Clearly the linear model still performs well, but the quadratic model is hopeless outside the orriginal range. This is because when we fitted the models, we had too many parameters (3) compared to the number of observations (4).


Edit: To address the query in the comments to this answer, about a model that does not include higher order terms.

The situation is the same: If the number of parameters approaches the number of observations, the model will be overfitted. With no higher order terms, this will occur when the number of variables / features in the model approaches the number of observations.

Again we can demonstrate this easily with a simulation:

Here we simulate random data data from a normal distribution, such that we have 7 observations and 5 variables / features:

> set.seed(1)
> n.var <- 5
> n.obs <- 7
> 
> dt <- as.data.frame(matrix(rnorm(n.var * n.obs), ncol = n.var))
> dt$Y <- rnorm(nrow(dt))
> 
> lm(Y ~ . , dt) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  -0.6607     0.2337  -2.827    0.216
V1            0.6999     0.1562   4.481    0.140
V2           -0.4751     0.3068  -1.549    0.365
V3            1.2683     0.3423   3.705    0.168
V4            0.3070     0.2823   1.087    0.473
V5            1.2154     0.3687   3.297    0.187

Residual standard error: 0.2227 on 1 degrees of freedom
Multiple R-squared:  0.9771,    Adjusted R-squared:  0.8627 

We obtain an adjusted R-squared of 0.86 which indicates excellent model fit. On purely random data. The model is severely overfitted. By comparison if we double the number of obervations to 14:

> set.seed(1)
> n.var <- 5
> n.obs <- 14
> dt <- as.data.frame(matrix(rnorm(n.var * n.obs), ncol = n.var))
> dt$Y <- rnorm(nrow(dt))
> lm(Y ~ . , dt) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)  
(Intercept) -0.10391    0.23512  -0.442   0.6702  
V1          -0.62357    0.32421  -1.923   0.0906 .
V2           0.39835    0.27693   1.438   0.1883  
V3          -0.02789    0.31347  -0.089   0.9313  
V4          -0.30869    0.30628  -1.008   0.3430  
V5          -0.38959    0.20767  -1.876   0.0975 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.7376 on 8 degrees of freedom
Multiple R-squared:  0.4074,    Adjusted R-squared:  0.03707 
F-statistic:   1.1 on 5 and 8 DF,  p-value: 0.4296

..adjusted R squared drops to just 0.037

$\endgroup$
9
  • 6
    $\begingroup$ I don't agree that "models with a degree of more than 1 have high chances of being overfitted". They have a highER chance of being overfitted compared to simpler models. As for models with a "degree of 1", if you are talking about a situation with only 1 explanatory variable, then overfitting would only occur is the number of observations is 2, in which case the model would be degenerate anyway. You could also argue that if the number of observations was 3 then it would also be overfitted. Overfitting can be a relative term. $\endgroup$ Aug 27 '20 at 11:18
  • 5
    $\begingroup$ If you are talking about several explanatory variables, then overfitting will occur when the number of explanatory variables approaches the number of observations, which is exactly the same as the situation in my answer (too many parameters compared the number of observations) $\endgroup$ Aug 27 '20 at 11:19
  • 3
    $\begingroup$ I think it would be more instructive to have the examples with degree 1 polynomials and e.g. 3 data points with substantial noise vs. 10 data points with same noise. $\endgroup$ Aug 27 '20 at 23:37
  • 3
    $\begingroup$ @CarlLeth by having insufficient observations or too many variables $\endgroup$ Aug 28 '20 at 15:22
  • 2
    $\begingroup$ @CarlLeth: degree 1 model too complex? Here are less complex possibilities: a degree 0 model or less variates: a degree 1 model in 1000d (with 1000 variates) is quite complex after all. Dropping variates btw. is like saying the model should be degree 0 for the dropped variates. BTW, you can even have too few data points to get a reliable estimate of the grand mean (degree 0 model). $\endgroup$ Aug 28 '20 at 18:13
4
$\begingroup$

Overfitting happens when the model performs well on the train data but doesn't do well on the test data. This is because the best fit line by your linear regression model is not a generalized one. This might be due to various factors. Some of the common factors are

  • Outliers in the train data.
  • Train and Test data are from different distributions.

So before building the model ensure that you have checked up on these factors to get a generalized model.

$\endgroup$
4
  • 4
    $\begingroup$ +1. To the downvoter: Give this user that's completely brand new to the SE network, a bit of a break. Edit their answer or write a comment explaining why you don't like their answer. Don't just downvote. Another option is to leave a negative comment without voting at all. This doesn't hurt the user as much, and gives you even more of a voice against the user. $\endgroup$ Aug 27 '20 at 18:07
  • 3
    $\begingroup$ (+1) Fully agree with @user1271772 $\endgroup$ Aug 27 '20 at 20:04
  • 3
    $\begingroup$ Overfitting is one of the reasons the model performs well on the train data but poorly on the test data, but not the only reason. The two examples you gave are examples of other reasons (not overfitting). Training outliers by themselves could affect your model performance, but it's not called overfitting unless you try too hard to have your model perform well on those outliers. Having different distributions doesn't mean your overfitting, it just means your data is bad. Building a good model from bad data is not overfitting. $\endgroup$
    – NotThatGuy
    Aug 28 '20 at 9:33
  • 3
    $\begingroup$ @user1271772 Are you implying that you upvoted simply because the user is new? You should vote based on what is posted, not who posted it. But yes, commenting to explain what's wrong with a post is generally helpful (even if such comments are often deleted by mods on a number of sites). $\endgroup$
    – NotThatGuy
    Aug 28 '20 at 9:45
2
$\begingroup$

Large number of parameters compared to data points

In general, one aspect of overfitting is trying to "invent information out of knowthing" when you want to determine a comparably large number of parameters from a limited amount of actual evidence data points.

For a simple linear regression y = ax + b there are two parameters, so for most sets of data it would be underparametrised, not overparametrised. However, let's look at the (degenerate) case of only two data points. In that situation you can always find a perfect linear regression solution - however, is that solution necessarily meaningful? Possibly not. If you treat the linear regression of two data points as a sufficient solution, that would be a prime example of overfitting.

Here is a nice example of overfitting with a linear regression by Randall Munroe of xkcd fame that illustrates this issue:

Extrapolating

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.