I have read that we make the following assumption for linear regression:
1. Linearity (correct functional form)
2. Constant error variance (homoskedasticity)
3. Independent error terms (no autocorrelation)
4. Normality of error terms
5. No multicollinearity
6. Exogeneity (no omitted variable bias)

So are these assumptions specific to Linear Regression or applicable for all types of regression techniques like Support Vector Regression, Lasso and Ridge regression, Stepwise regression etc.

  • 3
    $\begingroup$ These are typical assumptions in OLS for finding a minimum-variance unbiased estimator of the parameters and performing inference (confidence intervals, p-values) on the parameters. These assumptions are not, however, necessary for other forms of regression. Lasso, for instance, will tend to find groups of correlated predictors and give nonzero parameter estimates for only one variable per group. This doesn't mean that Lasso requires correlated predictors, but Lasso tolerates correlated predictors. $\endgroup$ – Dave Mar 11 '20 at 15:10
  • $\begingroup$ @Dave, +1 for most of it, but I'm not sure whether Lasso will tend to zero out all but one from a correlated group? At least, in the case of an actual duplicated column, lasso will split the coefficient evenly across the two. $\endgroup$ – Ben Reiniger Mar 11 '20 at 16:27
  • 1
    $\begingroup$ @BenReiniger You may be right about what happens when Lasso encounters duplicated columns, but I'd still say that it's the case that Lasso tends to zero-out all but one of a group of predictors with tight correlation. That may not always happen, but it does tend to. $\endgroup$ – Dave Mar 11 '20 at 16:31

These are not (direct) assumptions for linear regression. But rather for OLS-Ordinary Least Squares which is widely used to estimate the parameter of a linear regression model. OLS estimators minimize the sum of the squared errors (a difference between observed values and predicted values).

In order to guarantee finding best possible parameters, we make these assumptions (as with every optimisation)

But just because they are not present that does not mean that linear regression wont work.

Regarding your other models: If they use the same optimisation, than they have same assumptions.

  • 1
    $\begingroup$ This is not correct. We do not need normal errors for the Gauss-Markov theorem to apply and give is the best linear unbiased estimator of the parameter vector. Normal error terms is an assumption for inference, not for prediction, which is where the question focuses. $\endgroup$ – Dave Mar 11 '20 at 16:40
  • $\begingroup$ Yes that is true. Normality is not assumed on the error terms, but 3 other assumptions of the error term. Did not read carefully. $\endgroup$ – Noah Weber Mar 11 '20 at 16:48
  • 1
    $\begingroup$ Gauss-Markov does not even require independent error terms. They just have to be uncorrelated. My point is that we can relax a lot of typical assumptions if the goal is prediction rather than parameter inference. $\endgroup$ – Dave Mar 11 '20 at 16:51
  • $\begingroup$ Linear regression is an algorithm and it has 2 methods to find the best fit line OLS and Gradient Descent. So is it that these assumptions are not applicable when I run linear regression using Gradient Descent? $\endgroup$ – learnToCode Mar 12 '20 at 6:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.