2
$\begingroup$

When evaluating the output from a linear/ridge regression model, I have taken the residuals between the predicted and test data. This gives me a normal distribution when I plot this data as a histogram.

Please can one explain why this is expected and important?

Improve this question
$\endgroup$
1
  • $\begingroup$ The situation is much more complicated. You will never have perfect normality even if the plot looks good, but your regression may well be good even if you don't have normality (potentially clearly visibly so). But then it is worthwhile checking plots because some things are problematic. See my answers here for more details: stats.stackexchange.com/questions/538561/… $\endgroup$
    – Christian Hennig
    Commented Jun 29, 2023 at 21:21

2 Answers 2

Reset to default
1
$\begingroup$

Let's take a step back and think: if my predictions are good, how should they compare to the ground truth?

  1. The predictions should be close to the ground truth (well, which is why we do prediction in first place). In other words, we want most residues to be small.
  2. Usually, the drawback of underestimate is same as overestimate, so it is desirable that our predictions are not biased in either way (not often over- nor under-estimate). Translate to residues, this means having about half of the residues positive, and the other half negative.

Now put them together, what does the residues' distribution look like? It is 1. having most values appear near 0 (small); and 2. roughly symmetric (not skewed). This gives us a bell-shape thing which looks like a normal distribution.

Improve this answer
$\endgroup$
2
  • $\begingroup$ There are many non-normal distributions that fit the same description. Also, for any given position of the predictor, the distribution may in fact be skew, in whcih case there is no way to achieve a non-skew residual distribution, but a skew one may be just fine if it appropriately balances few large deviations on one side against many small deviations on the other side. $\endgroup$
    – Christian Hennig
    Commented Jun 30, 2023 at 9:51
  • $\begingroup$ @ChristianHennig 1. I didn't say it is a normal distribution, just looks like normally distributed. 2. true that the vanilla residue can be skewed, but after weighted by cost it should be balance - well, unless the cost function favors skewness, which is not unusual in many business application. $\endgroup$
    – lpounng
    Commented Jul 2, 2023 at 7:35
1
$\begingroup$

If you want a statistically significant result, I suggest you to verify the normality of the residuals distribution using the QQ-plot - you can find more informations about that here - or the Shapiro-Wilk test. To answer your question, you cannot make inference on a regression model if this assumption is not verified, because the estimators (e.g. the ones used for regression coefficients estimation) are unbiased only under THAT assumption. Since usually the aim of a linear regression is its use in statistical inference, it's important to verify that every time.

Improve this answer
$\endgroup$
3
  • $\begingroup$ You can't verify normality as no real data are ever properly continuous (they all come rounded to a limited number of digits), and hardly any real data allow for an unlimited value range. No real data are ever normal. The best you can ever do is exclude certain problematic issues such as outliers and strong skewness. See stats.stackexchange.com/questions/538561/… $\endgroup$
    – Christian Hennig
    Commented Jun 29, 2023 at 21:25
  • 1
    $\begingroup$ More, from other authors, here: stats.stackexchange.com/questions/2492/… $\endgroup$
    – Christian Hennig
    Commented Jun 29, 2023 at 21:46
  • $\begingroup$ You're right. With "verify normality" I meant you should see if the residuals have a distribution that tend to a Normal distribution, obviously it can't be exactly normal because if that were the case you would have a statistically perfect model that fully explains the variations in the dependent variable. $\endgroup$
    – PurpleCat
    Commented Jun 30, 2023 at 12:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.