3
$\begingroup$

I have read that we make the following assumption for linear regression:
1. Linearity (correct functional form)
2. Constant error variance (homoskedasticity)
3. Independent error terms (no autocorrelation)
4. Normality of error terms
5. No multicollinearity
6. Exogeneity (no omitted variable bias)

So are these assumptions specific to Linear Regression or applicable for all types of regression techniques like Support Vector Regression, Lasso and Ridge regression, Stepwise regression etc.

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ These are typical assumptions in OLS for finding a minimum-variance unbiased estimator of the parameters and performing inference (confidence intervals, p-values) on the parameters. These assumptions are not, however, necessary for other forms of regression. Lasso, for instance, will tend to find groups of correlated predictors and give nonzero parameter estimates for only one variable per group. This doesn't mean that Lasso requires correlated predictors, but Lasso tolerates correlated predictors. $\endgroup$
    – Dave
    Mar 11, 2020 at 15:10
  • $\begingroup$ @Dave, +1 for most of it, but I'm not sure whether Lasso will tend to zero out all but one from a correlated group? At least, in the case of an actual duplicated column, lasso will split the coefficient evenly across the two. $\endgroup$
    – Ben Reiniger ♦
    Mar 11, 2020 at 16:27
  • 1
    $\begingroup$ @BenReiniger You may be right about what happens when Lasso encounters duplicated columns, but I'd still say that it's the case that Lasso tends to zero-out all but one of a group of predictors with tight correlation. That may not always happen, but it does tend to. $\endgroup$
    – Dave
    Mar 11, 2020 at 16:31

2 Answers 2

Reset to default
1
$\begingroup$

These are not (direct) assumptions for linear regression. But rather for OLS-Ordinary Least Squares which is widely used to estimate the parameter of a linear regression model. OLS estimators minimize the sum of the squared errors (a difference between observed values and predicted values).

In order to guarantee finding best possible parameters, we make these assumptions (as with every optimisation)

But just because they are not present that does not mean that linear regression wont work.

Regarding your other models: If they use the same optimisation, than they have same assumptions.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ This is not correct. We do not need normal errors for the Gauss-Markov theorem to apply and give is the best linear unbiased estimator of the parameter vector. Normal error terms is an assumption for inference, not for prediction, which is where the question focuses. $\endgroup$
    – Dave
    Mar 11, 2020 at 16:40
  • $\begingroup$ Yes that is true. Normality is not assumed on the error terms, but 3 other assumptions of the error term. Did not read carefully. $\endgroup$
    – Noah Weber
    Mar 11, 2020 at 16:48
  • 1
    $\begingroup$ Gauss-Markov does not even require independent error terms. They just have to be uncorrelated. My point is that we can relax a lot of typical assumptions if the goal is prediction rather than parameter inference. $\endgroup$
    – Dave
    Mar 11, 2020 at 16:51
  • $\begingroup$ Linear regression is an algorithm and it has 2 methods to find the best fit line OLS and Gradient Descent. So is it that these assumptions are not applicable when I run linear regression using Gradient Descent? $\endgroup$
    – learnToCode
    Mar 12, 2020 at 6:10
  • $\begingroup$ @learnToCode Gradient descent is just a way of calculating the solution. For OLS linear regression, we get the solution by solving the calculus and getting $\hat\beta_{OLS} = (X^TX)^{-1}X^Ty$. For other models (such as neural networks) or linear regression estimation techniques other than OLS (such as LASSO), the calculus cannot be solved in general, so we resort to numerical methods like gradient descent. $\endgroup$
    – Dave
    Apr 25, 2022 at 18:23
0
$\begingroup$

To some extent, I disagree with every one of these.

  1. Nonlinear regression models (e.g., SVM) do not assume a linear functional form, and nonlinear basis functions (e.g., polynomials or splines) can allow linear models to fit trends that have curvature.

  2. Constant error variance could be useful for certain forms of inference, but it is hardly necessary for predictions.

  3. Time series models handle autocorrelation.

  4. Error term normality means that the OLS solution for a linear model corresponds to maximum likelihood estimation of the regression weights, and this is nice for doing inferences with the weights or with nested models. However, such inferences are pretty robust to deviations from normality (especially with large sample sizes). Further, other regression models like quantile regression assume a different error distribution in order to correspond to maximum likelihood estimation. In particular, quantile regression at the median (minimizing MAE) corresponds to maximum likelihood estimation for a Laplace-distributed error term.

  5. Even in OLS regression, there is no assumption of a lack of feature multicollinearity.

  6. Depending on the complexity of the model, my assumption might be that I have missed an important feature. If I can produce a useful model despite this, good for me.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.