I'm just starting to learn about linear regressions and was wondering why it is that we opt to minimize the sum of squared errors. I understand the squaring helps us balance positive and negative individual errors (so say e1 = -2 and e2 = 4, we'd consider them as both regular distances of 2 and 4 respectively before squaring them), however, I wonder why we don't deal with minimizing the absolute value rather than the squares. If you square it, e2 has a relatively higher individual contribution to minimize than e1 compared to just the absolutely values (and do we want that?). I also wonder about decimal values. For instance, say we have e1 = 0.5 and e2 = 1.05, e1 will be weighted less when squared because 0.25 is less than 0.5 and e2 will be weighted more. Lastly, there is the case of e1 = 0.5 and e2 = 0.2. E1 is further away to start, but when you square it 0.25 is compared with 0.4. Anyway, just wondering why we do sum of squares Erie minimization rather than absolute value.
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ en.m.wikipedia.org/wiki/Least_absolute_deviations $\endgroup$– SmallChessSep 29, 2017 at 23:49
-
$\begingroup$ $0.5^2=0.25 > 0.04 = 0.2^2$ Monotonic transformations preserve order. $\endgroup$– DaveSep 20, 2020 at 14:52
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
Simple google search on "stats why regression not absolute difference" would give you good answers. Try it yourself!
I can quickly summarise:
- Your regression parameters are solutions to the maximum likelihood optimisation. That involves derivative, but absolute difference doesn't have a derivative at zero. There's no unique solution for least absolute regression.
- Least absolute regression is an alternative to the regular sum of squares regression, commonly classified as one of the robust statistical methods.
- You'd prefer least absolute regression if you care about outliers, otherwise the regular regression is generally better.
You might want to read about L1 vs L2:
https://stats.stackexchange.com/questions/45643/why-l1-norm-for-sparse-models
answered Sep 30, 2017 at 12:42
-
$\begingroup$ Great! Thank you. I tried googling something like, "why do we use squares instead of absolute value in linear regression," but maybe that was too specific. Thank you for the information! I can do some reading to let it sink in a bit more. $\endgroup$– stk1234Sep 30, 2017 at 13:40
-
$\begingroup$ @Snaggletooth Please consider to accept the answer if it helps you. $\endgroup$ Sep 30, 2017 at 13:40
-
$\begingroup$ Done! New to this forum so didn't see that :) $\endgroup$– stk1234Sep 30, 2017 at 13:47
$\begingroup$
$\endgroup$
A question similar to this has already been asked on Cross Validated. See:
Why squared residuals instead of absolute residuals in OLS estimation? and
Why square the difference instead of taking the absolute value in standard deviation?.
The former is actually a duplicate question from the latter.
You may also benefit from answer to this post
