# The cross-entropy error function in neural networks

In the MNIST For ML Beginners they define cross-entropy as

$$H_{y'} (y) := - \sum_{i} y_{i}' \log (y_i)$$

$y_i$ is the predicted probability value for class $i$ and $y_i'$ is the true probability for that class.

## Question 1

Isn't it a problem that $y_i$ (in $\log(y_i)$) could be 0? This would mean that we have a really bad classifier, of course. But think of an error in our dataset, e.g. an "obvious" 1 labeled as 3. Would it simply crash? Does the model we chose (softmax activation at the end) basically never give the probability 0 for the correct class?

## Question 2

I've learned that cross-entropy is defined as

$$H_{y'}(y) := - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log (1-y_i)})$$

What is correct? Do you have any textbook references for either version? How do those functions differ in their properties (as error functions for neural networks)?

One way to interpret cross-entropy is to see it as a (minus) log-likelihood for the data $$y_i'$$, under a model $$y_i$$.

Namely, suppose that you have some fixed model (a.k.a. "hypothesis"), which predicts for $$n$$ classes $$\{1,2,\dots, n\}$$ their hypothetical occurrence probabilities $$y_1, y_2,\dots, y_n$$. Suppose that you now observe (in reality) $$k_1$$ instances of class $$1$$, $$k_2$$ instances of class $$2$$, $$k_n$$ instances of class $$n$$, etc. According to your model the likelihood of this happening is: $$P[data|model] := y_1^{k_1}y_2^{k_2}\dots y_n^{k_n}.$$ Taking the logarithm and changing the sign: $$-\log P[data|model] = -k_1\log y_1 -k_2\log y_2 - \dots -k_n\log y_n = -\sum_i k_i \log y_i$$ If you now divide the right-hand sum by the number of observations $$N = k_1+k_2+\dots+k_n$$, and denote the empirical probabilities as $$y_i'=k_i/N$$, you'll get the cross-entropy: $$-\frac{1}{N} \log P[data|model] = -\frac{1}{N}\sum_i k_i \log y_i = -\sum_i y_i'\log y_i =: H(y', y)$$

Furthermore, the log-likelihood of a dataset given a model can be interpreted as a measure of "encoding length" - the number of bits you expect to spend to encode this information if your encoding scheme would be based on your hypothesis.

This follows from the observation that an independent event with probability $$y_i$$ requires at least $$-\log_2 y_i$$ bits to encode it (assuming efficient coding), and consequently the expression $$-\sum_i y_i'\log_2 y_i,$$ is literally the expected length of the encoding, where the encoding lengths for the events are computed using the "hypothesized" distribution, while the expectation is taken over the actual one.

Finally, instead of saying "measure of expected encoding length" I really like to use the informal term "measure of surprise". If you need a lot of bits to encode an expected event from a distribution, the distribution is "really surprising" for you.

With those intuitions in mind, the answers to your questions can be seen as follows:

• Question 1. Yes. It is a problem whenever the corresponding $$y_i'$$ is nonzero at the same time. It corresponds to the situation where your model believes that some class has zero probability of occurrence, and yet the class pops up in reality. As a result, the "surprise" of your model is infinitely great: your model did not account for that event and now needs infinitely many bits to encode it. That is why you get infinity as your cross-entropy.

To avoid this problem you need to make sure that your model does not make rash assumptions about something being impossible while it can happen. In reality, people tend to use sigmoid or "softmax" functions as their hypothesis models, which are conservative enough to leave at least some chance for every option.

If you use some other hypothesis model, it is up to you to regularize (aka "smooth") it so that it would not hypothesize zeros where it should not.

• Question 2. In this formula, one usually assumes $$y_i'$$ to be either $$0$$ or $$1$$, while $$y_i$$ is the model's probability hypothesis for the corresponding input. If you look closely, you will see that it is simply a $$-\log P[data|model]$$ for binary data, an equivalent of the second equation in this answer.

Hence, strictly speaking, although it is still a log-likelihood, this is not syntactically equivalent to cross-entropy. What some people mean when referring to such an expression as cross-entropy is that it is, in fact, a sum over binary cross-entropies for individual points in the dataset: $$\sum_i H(y_i', y_i),$$ where $$y_i'$$ and $$y_i$$ have to be interpreted as the corresponding binary distributions $$(y_i', 1-y_i')$$ and $$(y_i, 1-y_i)$$.

• Can you provide a source where they define $y′i=\frac{ki}{N}$? Here they define it as a one-hot distribution for the current class label. What is the difference? Jun 22, 2016 at 7:47
• In the MNIST TensorFlow tutorial they define it in terms of one-hot vectors as well. Jun 22, 2016 at 9:32
• @Alex This may need longer explanation to understand properly - read up on Shannon-Fano codes and relation of optimal coding to the Shannon entropy equation. To dumb things down, if an event has probability 1/2, your best bet is to code it using a single bit. If it has probability 1/4, you should spend 2 bits to encode it, etc. In general, if your set of events has probabilities of the form 1/2^k, you should give them lengths k - this way your code will approach the Shannon optimal length.
– KT.
Aug 21, 2017 at 9:55
• If your event probabilities are not powers of 2, you will still get a near-optimal code if each event is coded in ceil(-log2(p)) bits (and such a code exists). Moreover, if you can encode sequences of events, their probabilities will be approaching some powers of 2 and your multi-event code will approach the optimal one if you spend close to -log2(p) bits per event with probability p.
– KT.
Aug 21, 2017 at 9:56
• (+1) This answer is really amazing! Great explanation for technical and intuitive understanding. Apr 12, 2020 at 18:38

The first logloss formula you are using is for multiclass log loss, where the $i$ subscript enumerates the different classes in an example. The formula assumes that a single $y_i'$ in each example is 1, and the rest are all 0.

That means the formula only captures error on the target class. It discards any notion of errors that you might consider "false positive" and does not care how predicted probabilities are distributed other than predicted probability of the true class.

Another assumption is that $\sum_i y_i = 1$ for the predictions of each example. A softmax layer does this automatically - if you use something different you will need to scale the outputs to meet that constraint.

## Question 1

Isn't it a problem that the $y_i$ (in $log(y_i)$) could be 0?

Yes that can be a problem, but it is usually not a practical one. A randomly-initialised softmax layer is extremely unlikely to output an exact 0 in any class. But it is possible, so worth allowing for it. First, don't evaluate $log(y_i)$ for any $y_i'=0$, because the negative classes always contribute 0 to the error. Second, in practical code you can limit the value to something like log( max( y_predict, 1e-15 ) ) for numerical stability - in many cases it is not required, but this is sensible defensive programming.

## Question 2

I've learned that cross-entropy is defined as $H_{y'}(y) := - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log (1-y_i)})$

This formulation is often used for a network with one output predicting two classes (usually positive class membership for 1 and negative for 0 output). In that case $i$ may only have one value - you can lose the sum over $i$.

If you modify such a network to have two opposing outputs and use softmax plus the first logloss definition, then you can see that in fact it is the same error measurement but folding the error metric for two classes into a single output.

If there is more than one class to predict membership of, and the classes are not exclusive i.e. an example could be any or all of the classes at the same time, then you will need to use this second formulation. For digit recognition that is not the case (a written digit should only have one "true" class)

• Note there is some ambiguity in the presentation of the second formula - it could in theory assume just one class and $i$ would then enumerate the examples. Dec 10, 2015 at 16:24
• I'm sorry, I've asked something different than what I wanted to know. I don't see a problem in $\log(y_i) = 0$, but in $y_i = 0$, because of $\log(y_i)$. Could you please adjust your answer to that? Dec 17, 2015 at 8:47
• @NeilSlater if the classes were not mutually exclusive, the output vector for each input may contain more than one 1, should we use the second formula? Feb 28, 2018 at 13:15
• @Media: Not really. You want to be looking at things such as hierarchical classification though . . . Feb 28, 2018 at 15:38
• @Javi: In the OP's question $y'_i$ is the ground truth, thus usually 0 or 1. It is $y_i$ that is the softmax output. However $y_i$ can end up zero in practice due to floating point rounding. This does actually happen. Feb 1, 2019 at 15:46

Given $$y_{true}$$, you want to optimize your machine learning method to get the $$y_{predict}$$ as close as possible to $$y_{true}$$.

## First question:

Above answer has explained the background of your first formula, the cross entropy defined in information theory.

### From a opinion other than information theory:

you can examine yourself that first formula does not have penalty on false-positiveness(truth is false but your model predict that it is right), while the second one has penalty on false-positiveness. Therefore, the choice of first formula or second, will affect your metrics(aka what statistic quantity you would like to use to evaluate your model).

### In layman word:

If you want to accept almost all good people to be your friend but willing to accept some bad people become your friend, then use first formula for criterion.

If you want to punish yourself accepting some bad people to be your friend,but at the same time your good-people accepting rate might be lower than the first condition, then use second formula.

While, I guess most of us are critical and would like to choose the second one(so as many ML package assume what is cross entropy).

## Second question:

Cross entropy per sample per class: $$-y_{true}\log{(y_{predict})}$$

Cross entropy for whole datasets whole classes: $$\sum_i^n \sum_k^K -y_{true}^{(k)}\log{(y_{predict}^{(k)})}$$

### Thus, when there are only two classes (K = 2), you will have the second formula.

Those issues are handled by the tutorial's use of softmax.

For 1) you're correct that softmax guarantees a non-zero output because it exponentiates it's input. For activations that do not give this guarantee (like relu), it's simple to add a very small positive term to every output to avoid that problem.

As for 2), they aren't the same obviously, but I the softmax formulation they gave takes care of the the issue. If you didn't use softmax, this would cause you to learn huge bias terms that guess 1 for every class for any input. But since they normalize the softmax across all classes, the only way to maximize the output of the correct class is for it to be large relative to the incorrect classes.

• "you're correct that softmax guarantees a non-zero output" - I know that this is theoretically the case. In reality, can it happen that (due to numeric issues) this becomes 0? Dec 10, 2015 at 14:30
• Good question. I assume it's perfectly possible for the exponentiation function to output 0.0 if your input is too small for the precision of your float. However I'd guess most implementations do add the tiny positive term to guarantee non-zero input. Dec 10, 2015 at 14:50

Isn't it a problem that $$y_i$$ (in $$\log(y_i)$$) could be 0?

Yes it is, since $$\log(0)$$ is undefined, but this problem is avoided using $$\log(y_i + \epsilon)$$ in practice.

What is correct?
(a) $$H_{y'} (y) := - \sum_{i} y_{i}' \log (y_i)$$ or
(b) $$H_{y'}(y) := - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log(1-y_i)})$$?

(a) is correct for multi-class prediction (it is actually a double summation), (b) is the same as (a) for two-class prediction. Both are cross-entropy.

### Example:

Suppose each training data $$x_i$$ has label $$c_i' \in \{0, 1\}$$, and model predicts $$c_i \in [0, 1]$$.

For 5 data points, true label $$c_i'$$ and model prediction $$c_i$$ are:

$$(c_i', c_i)=\{(0, 0.1), (0, 0.4), (0, 0.8), (1, 0.8), (1, 0.2)\}$$ (1),

Define vectors $$y_i'$$ and $$y_i$$ as

• $$y_{ik}':=1$$ if $$c_i'=k$$, and $$:=0$$ otherwise,

• $$y_{ik}:=p(k|x_i)$$ is the probability of $$x_i$$ belonging to class $$k$$, which is estimated by model.

Example (1) in $$(y_i', y_i)$$ notation turns into:

$$(y_i', y_i)=\{([1, 0], [0.9, 0.1]),$$ $$([1, 0], [0.6, 0.4]),$$ $$([1, 0], [0.2, 0.8]),$$ $$([0, 1], [0.2, 0.8]),$$ $$([0, 1], [0.8, 0.2])\}$$,

Both (a) and (b) are calculated as:

$$H_{y'}(y)=-1/5([log(0.9)+log(0.6) + log(0.2)]_{c_i=0} + [log(0.8) + log(0.2)]_{c_i=1}) = 0.352$$

### Derivation:

Suppose there is multiple classes $$1$$ to $$K$$.
For training point $$(x_i, c_i')$$, $$c_i' = k$$ is equivalent to $$y_i'=[0,..,1,0,..]$$ which is 1 in $$k^{th}$$ position and 0 elsewhere. When $$y_{ik}'=1$$, we want model's output $$y_{ik}=p(k|x_i)$$ to be close to 1. Therefore, loss of $$(x_i, k)$$ can be defined as $$-log(y_{ik})$$, which gives $$y_{ik} \rightarrow 1 \Rightarrow -log(y_{ik}) \rightarrow 0$$. Loss over all classes can be combined as:

$$L(y_i', y_i) = -\sum_{k=1}^{K}y_{ik}'log(y_{ik})$$.

When $$y_{ik}' = 1$$, loss of all other classes $$k' \neq k$$ is disabled as $$0log(y_{ik'})=0$$, so for example when true label is $$y_{im}'=1$$, loss would be:

$$L(y_i', y_i)=-log(y_{im})$$.

Final formula over all training points is:

$$H_{y'}(y)=-\sum_{(x_i, y_i')}\sum_{k=1}^{K}y_{ik}'log(y_{ik})$$.

For binary classification, we have $$y_{i0}' = 1 - y_{i1}'$$ (true labels) and $$y_{i0} = 1 - y_{i1}$$ (model predictions), therefore (a) can be rewritten as:

\begin{align*} H_{y'}(y)&=-\sum_{(x_i, y_i')}y_{i1}'log(y_{i1})+y_{i0}'log(y_{i0})\\ &=-\sum_{(x_i, y_i')}y_{i1}'log(y_{i1})+(1-y_{i1}')log(1-y_{i1}) \end{align*}

which is the same as (b).

Cross-entropy (a) over classes (one summation)

Cross-entropy (a) over classes is:

$$H_{y'}(y)=-\sum_{k=1}^{K}y_{k}'log(y_{k})$$,

This version cannot be used for the classification task. Lets reuse the data from the previous example:

$$(c_i', c_i)=\{(0, 0.1), (0, 0.4), (0, 0.8), (1, 0.8), (1, 0.2)\}$$

Empirical class probabilities are: $$y'_0 = 3/5 = 0.6$$, and $$y'_1 = 0.4$$,

Class probabilities estimated by model are: $$y_0 = 3/5 = 0.6$$, and $$y_1 = 0.4$$

(a) is calculated as: $$-y'_0logy_0 - y'_1logy_1 = - 0.6log(0.6) -0.4log(0.4) = 0.292$$.

Two data points $$(0, 0.8)$$ and $$(1, 0.2)$$ are miss-classified but $$y'_0$$ and $$y'_1$$ are estimated correctly!

If all 5 points where classified correctly as:
$$(c_i', c_i)=\{(0, 0.1), (0, 0.4), (0, \color{blue}{0.2}), (1, 0.8), (1, \color{blue}{0.8})\}$$ ,

(a) still remains the same, since $$y'_0$$ is again estimated as $$y_0=3/5$$.

Question 2 I've learned that cross-entropy is defined as...

These two means either two things.

1. The person who said you that has no idea about he/she is talking about
2. You incorrectly learned it.

In Mathematics Kullback-Leiber divergence(KL), Cross-Entropy(CE), Entropy(H) always mean only one thing, but the term Entropy unfortunately can vary from the scientific community. In any case, the good book on the subject "Information Theory is the book "Elements of Information Theory" by Thomas M. Cover, Joy A. Thomas." from 1991.

But if you don't want to understand mathematical properties (they maybe are not used by you) then you can check the definition from Wikipedia:

1. The function is defined in two discrete (or continuous distributions)
2. In the case of discrete distributions the summation happens not for all possible $$x=x(w_i)$$ values of random variables but only for which the two p.m.f. has a positive value of probabilities. (In math languages the summation happens over the intersection of the support of two functions)

How to make this computation numerical stable for such a situation when $$\exists i:p_i\sim 0, q_i \sim0$$ it's another question. And one way consider how $$p,q$$ are generated a-priori. For example in case of using some form of symmetric logistic transformation (name from STATs) or softmax (one of the name from Machine Learning) then it's nice to take into account log-sum-exp trick. References: https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss https://gregorygundersen.com/blog/2020/02/09/log-sum-exp/