3
$\begingroup$

Training a basic multilayer perceptron neural network boils down to minimizing some kind of error function. Often the sum of squared errors is chosen as a this error function, but where does this function come from?

I always thought this function was chosen because it makes sense intuitively. However, recently I learned that this is only partly true and there is more behind it.

Bishop wrote in one of his papers that the sum of squared errors function can be derived from the principle of maximum likelihood. Furthermore he wrote that the squared error therefore makes the assumption that the noise on the target value has a Gaussian distribution.

I am not sure what he means with that. How does the sum of squared errors relate to the maximum likelihood principle in the context of neural networks?

Improve this question
$\endgroup$
1
  • $\begingroup$ Can you specify which paper you are citing? $\endgroup$
    – Clumsy cat
    Apr 22, 2018 at 12:34

2 Answers 2

Reset to default
4
$\begingroup$

you can trace the squared error in statistics through multivariate calculus all the way to Pythagorus. You are basically calculating the 'effective length' of the error, the hypotenuse, among errors from multiple variables $(X_1 - X_2)^2 + (Y_1 - Y_2)^2 + ...$ like in a triangle.

But where did the square root go?

Somebody realized that calculating roots of multiple variables over multiple iterations is computationally very expensive. So they decided to drop it. Checkout the squared Euclidean distance here for more details

How would a cubic error or a logarithmic error affect the outcome?

It just takes more time to converge because they are not as accurate. But we do see logarithmic errors over squares such as logistic regression where it is more optimal

All in all it is a simple case of optimization

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ I don't believe that computational expense is the main reason why square root is not taken over sum-of-squared errors. (I don't have any evidence one way or the other.) After all, you only need to perform the square root operation once for all error values. Instead, I believe that sum-of-squared errors serves a different decision criterion. Squaring the errors puts more weight on large errors, compared to linear or log functions. Therefore, minimizing sum-of-squares error serves this decision criterion: "Favor the model that has the least aggregate large errors." $\endgroup$
    – MrMeritology
    Jun 7, 2015 at 20:28
  • 1
    $\begingroup$ if we are only looking to minimize large aggregate errors, a cubic error is better than a squared error is it not. I know I read it somewhere mentioning the reason as 'optimization technique' but can't recall where $\endgroup$
    – device_exec
    Jun 9, 2015 at 13:47
  • $\begingroup$ Whilst geometrically it's a pythagorus distance, in ANN we're just minimizing an algebraic function, and I don't see properties that $a²+b²+…+z²$ exhibits and $a + b + … + z$ does not, except of MrMeritology's comment. Then perhaps would it worth just using a sum $a+b+… +z$ instead? $\endgroup$
    – Hi-Angel
    Feb 24, 2017 at 10:27
2
$\begingroup$

Your reference of Bishop is not entirely accurate. What he states in the paper you linked is

It should be noted that the standard sum-of-squares error, introduced here from a heuristic viewpoint, can be derived from the principle of maximum likelihood on the assumption that the noise on the target data has a Gaussian distribution [references cited]. Even when this assumption is not satisfied, however, the sum-of-squares error function remains of great practical importance.

The important point with regard to your question is that there is no inherent assumption that there is Gaussian noise when training a Multilayer Perceptron (MLP). Therefore, for an MLP, the sum-of-squares error function is not derived from the principle of maximum likelihood.

For example, consider training an MLP to learn the XOR function. There are four pairs of inputs with corresponding outputs but there is no noise in the data. Yet the sum-of-squares error is still applicable.

The relevance of using sum-of-squares for neural networks (and many other situations) is that the error function is differentiable and since the errors are squared, it can be used to reduce or minimize the magnitudes of both positive and negative errors.

Improve this answer
$\endgroup$
4
  • $\begingroup$ It is not just that squaring the errors has the effect of treating positive and negative errors equally, because abs( ) would have the same effect. Squaring the errors puts more weight on large errors, compared to linear or log functions. Therefore, minimizing sum-of-squares error serves this decision criteria: "Favor the model that has the least aggregate large errors." $\endgroup$
    – MrMeritology
    Jun 7, 2015 at 20:13
  • $\begingroup$ Yes, using squared errors puts greater emphasis on large errors but abs() has other issues, like the fact that its derivative is discontinuous at 0. And regarding your stated decision criterion, wouldn't that be even better met by $E^4$ or $E^6$? I don't have a citation but I suspect the real reason squared errors are used is related to the quote in the original question: errors tend to be Gaussian and minimizing the squared errors provides the maximum likelihood estimate in that case. They also have the benefit of being easily differentiable, which makes them simple to use in practice. $\endgroup$
    – bogatron
    Jun 7, 2015 at 23:14
  • $\begingroup$ Good points. Yes, abs( ) is discontinuous at 0, but I'm not sure what difference that makes in empirical analysis. Yes, higher ordered error functions would weight large errors even more, and would support the decision criterion mentioned. If minimizing squared errors leads to max. likelihood estimate (assuming Gaussian), then I agree with that line of reasoning. $\endgroup$
    – MrMeritology
    Jun 8, 2015 at 1:28
  • 1
    $\begingroup$ @MrMeritology discontinuity at 0 wouldn't bother me either; but there is an other thing that makes (y-f(x))^2 a much better choice than abs(y-f(x)): Imagine you had two data points both at the same x (x,y1) and (x,y2). Then sum of squares error is minimal at the mean of y1 and y2 while the absolute error is the same anywhere between y1 and y2. And the mean feels like a good choice for me. $\endgroup$
    – jan-glx
    Sep 12, 2015 at 12:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.