When computing loss functions, people use $(target-actual)^2$. They sqaure it to prevent any negative loss. But we can even use $|(target-actual)|$ to prevent any negative loss. So, why do people prefer the first option more than the second?


Apart from the correct answers which you find in the comment section, you mention "the square [..] prevent any negative loss".

In principle you can also have a negative loss. The point is without the square, you have $(x-y) \neq (y-x)$ for $x \neq y$. In particular, the loss would not be symmetric and for $x = 0$, you have $(x-y) = -y$. So by increasing $y$ you decrease the loss. The loss would thus not be lower bounded so that there is no global minimum for the loss.

| improve this answer | |
  • $\begingroup$ You said 'In principle you can also have a negative loss', I don't think we can have negative losses because then the negative losses could cancel out with the positive losses. $\endgroup$ – Dhruv Agarwal Sep 23 at 13:22
  • $\begingroup$ If you minimize $\sum_{i} ||\mathrm{target}_{i}-\mathrm{actual}_{i}||_{2}^{2}-\theta_{i}$, where $\theta = \theta_{i} > 0$ is fixed. you will get the same optimal result and the loss can have negative values. $\endgroup$ – Graph4Me Consultant Sep 23 at 13:24
  • $\begingroup$ ok, may be replace "loss" with "objective function". But still my point is that without the square, computing $x-y$ is completely wrong (and this is not due to negative values) but as it is not measuring anything usefull. $\endgroup$ – Graph4Me Consultant Sep 23 at 13:41
  • $\begingroup$ For example, you could have a neural network that minimizes or maximizes the cosine similarity. In this case, the optimal objective value is either 1 or -1. $\endgroup$ – Graph4Me Consultant Sep 23 at 13:44

Not the answer you're looking for? Browse other questions tagged or ask your own question.