Confidence interval interpretation in linear regression when errors are not normally distributed

Question

I've read that "If the error distribution is significantly non-normal, confidence intervals may be too wide or too narrow" (source). So, can anyone elaborate on this? When are the confidence intervals narrow and when are they wide? Does it have anything to do with skewness?

Answer 1

OLS Model:

One of the assumptions behind OLS (aka linear regression) is homoskedasticity, namely:

$$ Var(u| x ) = \sigma^2.$$

Recall that the linear model is defined:

$$ y = X \beta + u, $$

where $u$ is the statistical error term. The error term (per OLS assumptions) need to have an expected value $E(u|x)=0$ (orthogonality condition) with variance $\sigma^2$, so that the error is distributed $u \sim (0,\sigma^2)$.

Heteroscedasticity:

In case the variance of $u$ is not "harmonic" and the assumption above is violated, we say that error terms are heteroscedastic. Heteroscedasticity does not (!) change the estimated coefficients, but it does affect the (estimated) standard errors and consequently the confidence bands.

The error variance is estimated by:

$$ \hat{\sigma}^2 = 1/(n-2) \sum{\hat{u}^2} .$$

The standard error (of coefficient $\beta$) is estimated by:

$$ se(\hat{\beta}) = \hat{\sigma} / (\sum{(x_i-\bar{x})^2})^{1/2}.$$

The assumption of homoskedasticity is required in order to get proper estimates of the error variance and the ("normal", in contrast to "robust", see below) standard errors. Standard errors in turn are used to calculate confidence bands. So in case you cannot trust the estimated standard errors, you can also not rely on the confidence bands.

The problem here ultimately is, that given heteroscedasticity, you cannot tell if some estimated coefficient is statistically significant or not. Significance here is defined (95% confidence) so that the confidence band of some estimated coefficient does not „cross“ zero (so is strictly positive or negative).

There are different options to deal with heteroscedasticity:

• The most common solution is to use "robust" standard errors. There are different versions of "robust" errors (HC1, HC2, HC3). They all have in common, that they aim at getting a "robust" estimate of the error variance. Most software allows you to calculate robust SE. Find an example for R here.
• Another alternative would be to estimate a "feasible generalised model" (FGLS) in which you first estimate the scedastic function (to get an idea of the distribution of errors) and you try to "correct" problems in the error distribution. However, this is not something you would use very often in practice. It is more an academic excercise.

Testing heteroscedasticity:

Usually, you would test if there is heteroscedasticity. You can look at the "residual vs. fitted plot" to get an idea of how the error terms are distributed.

However, a proper test can be done using the White or Breusch-Pagan Tests. Here is an example in R.

Answer 2

In case you might want to try another way to find confidence intervals, and in addition to the nice and rigourous answer by Peter, I would also consider a resampling method like bootstrapping as a robust way to find confidence intervals. One key advantage is that it does not assume any kind of distribution, being a distribution-free method to find your coefficients estimates.

In the case of finding a confidence interval for a linear regression, the steps would be:

1. Draw n random samples (with replacement) from your dataset, where n is the bootstrap sample size
2. Fit a linear regression on the bootstrap sample from step 1
3. Repeat steps 1 & 2 n_iters times, where n_iters will be the number of bootstrap samples and linear regressions made on them
4. Now that we have n_iters values for the linear regression coefficients, we can find the interval limits via the min, median and max percentiles (e.g. for a 95% CI: percentile 2.5, 50 and 97.5) to find the coefficient estimate together with the CI limits

Please mind the variability of the confidence intervals along the x-axis values, taking into account the sampling error of the coefficients estimates (good source of read: https://greenteapress.com/wp/think-stats-2e/)

The associated code of my example including the plot can be found here