# How is calculated the error with multiple output neurons in neural network?

Machine Learning books generally explains that the error calculated for a given sample $$i$$ is:

$$e_i = y_i - \hat{y_i}$$

Where $$\hat{y}$$ is the target output and $$y$$ is the actual output given by the network. So, a loss function $$L$$ is calculated:

$$L = \frac{1}{2N}\sum^{N}_{i=1}(e_i)^2$$

The above scenario is explained for a binary classification/regression problem. Now, let's assume a MLP network with $$m$$ neurons in the output layer for a multiclass classification problem (generally one neuron per class).

What does change in the equations above? Since we now have multiple outputs, both $$e_i$$ and $$y_i$$ should be a vector?

• I thought that every neuron would have its $L$. For instance, for neuron $m$: $L_m = \frac{1}{2N} \sum_{i=1}^{N}(e_{im})^2$ – heresthebuzz Oct 20 '20 at 21:42

## 1 Answer

You are mixing various concepts:

• $$L = \frac{1}{2N}\sum^{N}_{i=1}(e_i)^2$$ is used only for regression problem and not for binary classification because MSE fits very well when your target distribution is normal
• You can use the latter formula for binary classification but will works really bad because your target data distribution is a Bernoulli, not Normal. Remember that the choice of the right imply a prior assumption on the target data distribution. For this reason the right formula is binary crossentropy (aka negative log likelihood of a Bernoulli) $$L = - \sum_i y_i \log \hat{y_i} (1 - y_i) \log(1 - \hat{y_i})$$
• For multi classification problem there is a generalized formula of binary crossentropy which is called categorical crossentropy. If $$\hat{y}$$ is a vector of C element, one for each class and the true class $$y$$ is encoded as integer (e.g 0, 1, 2 ...) then the loss is $$L = - \sum_i \log(\hat{y_i}[y])$$
• Thanks for the explanation. I've seen some models built with cross-entropy (e.g. Keras and Tensorflow). However, classical books still use MSE as loss criteria... in that case, does it make sense if the error is a vector? – heresthebuzz Oct 20 '20 at 22:13
• Honestly I never seen any book that use mse for binary classification and if you have such book you should change book. If i can suggest one very well writter check this deeplearningbook.org . MSE it makes sense also on predicted vectors (a common case are images) just add a sum term for each element of the vector – Mikedev Oct 20 '20 at 22:18
• Thank you very much. – heresthebuzz Oct 20 '20 at 22:20