Machine Learning books generally explains that the error calculated for a given sample $i$ is:

$e_i = y_i - \hat{y_i}$

Where $\hat{y}$ is the target output and $y$ is the actual output given by the network. So, a loss function $L$ is calculated:

$L = \frac{1}{2N}\sum^{N}_{i=1}(e_i)^2$

The above scenario is explained for a binary classification/regression problem. Now, let's assume a MLP network with $m$ neurons in the output layer for a multiclass classification problem (generally one neuron per class).

What does change in the equations above? Since we now have multiple outputs, both $e_i$ and $y_i$ should be a vector?

  • $\begingroup$ I thought that every neuron would have its $L$. For instance, for neuron $m$: $L_m = \frac{1}{2N} \sum_{i=1}^{N}(e_{im})^2$ $\endgroup$ Oct 20, 2020 at 21:42

1 Answer 1


You are mixing various concepts:

  • $L = \frac{1}{2N}\sum^{N}_{i=1}(e_i)^2$ is used only for regression problem and not for binary classification because MSE fits very well when your target distribution is normal
  • You can use the latter formula for binary classification but will works really bad because your target data distribution is a Bernoulli, not Normal. Remember that the choice of the right imply a prior assumption on the target data distribution. For this reason the right formula is binary crossentropy (aka negative log likelihood of a Bernoulli) $$ L = - \sum_i y_i \log \hat{y_i} (1 - y_i) \log(1 - \hat{y_i}) $$
  • For multi classification problem there is a generalized formula of binary crossentropy which is called categorical crossentropy. If $\hat{y}$ is a vector of C element, one for each class and the true class $y$ is encoded as integer (e.g 0, 1, 2 ...) then the loss is $$ L = - \sum_i \log(\hat{y_i}[y]) $$
  • $\begingroup$ Thanks for the explanation. I've seen some models built with cross-entropy (e.g. Keras and Tensorflow). However, classical books still use MSE as loss criteria... in that case, does it make sense if the error is a vector? $\endgroup$ Oct 20, 2020 at 22:13
  • $\begingroup$ Honestly I never seen any book that use mse for binary classification and if you have such book you should change book. If i can suggest one very well writter check this deeplearningbook.org . MSE it makes sense also on predicted vectors (a common case are images) just add a sum term for each element of the vector $\endgroup$
    – Mikedev
    Oct 20, 2020 at 22:18
  • $\begingroup$ MSE in a classification problem is known as “Brier score” and has been used to train image classifiers. $\endgroup$
    – Dave
    Nov 22, 2021 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.