In the MNIST For ML Beginners they define cross-entropy as

$$H_{y'} (y) := - \sum_{i} y_{i}' \log (y_i)$$

$y_i$ is the predicted probability value for class $i$ and $y_i'$ is the true probability for that class.

Question 1

Isn't it a problem that $y_i$ (in $\log(y_i)$) could be 0? This would mean that we have a really bad classifier, of course. But think of an error in our dataset, e.g. an "obvious" 1 labeled as 3. Would it simply crash? Does the model we chose (softmax activation at the end) basically never give the probability 0 for the correct class?

Question 2

I've learned that cross-entropy is defined as

$$H_{y'}(y) := - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log (1-y_i)})$$

What is correct? Do you have any textbook references for either version? How do those functions differ in their properties (as error functions for neural networks)?

One way to interpret cross-entropy is to see it as a (minus) log-likelihood for the data $y_i'$, under a model $y_i$.

Namely, suppose that you have some fixed model (a.k.a. "hypothesis"), which predicts for $n$ classes $\{1,2,\dots, n\}$ their hypothetical occurrence probabilities $y_1, y_2,\dots, y_n$. Suppose that you now observe (in reality) $k_1$ instances of class $1$, $k_2$ instances of class $2$, $k_n$ instances of class $n$, etc. According to your model the likelihood of this happening is: $$ P[data|model] := y_1^{k_1}y_2^{k_2}\dots y_n^{k_n}. $$ Taking the logarithm and changing the sign: $$ -\log P[data|model] = -k_1\log y_1 -k_2\log y_2 - \dots -k_n\log y_n = -\sum_i k_i \log y_i $$ If you now divide the right-hand sum by the number of observations $N = k_1+k_2+\dots+k_n$, and denote the empirical probabilities as $y_i'=k_i/N$, you'll get the cross-entropy: $$ -\frac{1}{N} \log P[data|model] = -\frac{1}{N}\sum_i k_i \log y_i = -\sum_i y_i'\log y_i =: H(y', y) $$

Furthermore, a log-likelihood of a dataset given a model can be interpreted as a measure of "encoding length" - the number of bits you expect to spend to encode this information if your encoding scheme would be based on your hypothesis.

This follows from the observation that an independent event with probability $y_i$ requires at least $-\log_2 y_i$ bits to encode it (assuming efficient coding), and consequently the expression $$-\sum_i y_i'\log_2 y_i,$$ is literally the expected length of the encoding, where the encoding lengths for the events are computed using the "hypothesized" distribution, while the expectation is taken over the actual one.

Finally, instead of saying "measure of expected encoding length" I really like to use the informal term "measure of surprise". If you need a lot of bits to encode an expected event from a distribution, the distribution is "really surprising" for you.

With those intuitions in mind, the answers to your questions can be seen as follows:

  • Question 1. Yes. It is a problem whenever the corresponding $y_i'$ is nonzero at the same time. It corresponds to the situation where your model believes that some class has zero probablity of occurrence, and yet the class pops up in reality. As a result, the "surprise" of your model is infinitely great: your model did not account for that event and now needs infinitely many bits to encode it. That's why you get inifinity as your cross-entropy.

    To avoid this problem you need to make sure that your model does not make rash assumptions about something being impossible while it can happen. In reality people tend to use sigmoid or "softmax" functions as their hypothesis models, which are conservative enough to leave at least some chance for every option.

    If you use some other hypothesis model, it is up to you to regularize (aka "smooth") it so that it wouldn't hypothesize zeros where it should not have.

  • Question 2. In this formula one usually assumes $y_i'$ to be either $0$ or $1$, while $y_i$ is the model's probability hypothesis for the corresponding input. If you look close enough, you will see that it is simply a $-\log P[data|model]$ for binary data, an equivalent of the second equation in this answer.

    Hence, strictly speaking, although it is still a log-likelihood, this is not syntactically equivalent to cross-entropy. What some people mean when referring to such an expression as cross-entropy is that it is, in fact, a sum over over binary cross-entropies for individual points in the dataset: $$ \sum_i H(y_i', y_i), $$ where $y_i'$ and $y_i$ have to be interpreted as the corresponding binary distributions $(y_i', 1-y_i')$ and $(y_i, 1-y_i)$.

  • 1
    Can you provide a source where they define $y′i=\frac{ki}{N}$? Here they define it as a one-hot distribution for the current class label. What is the difference? – Lenar Hoyt Jun 22 '16 at 7:47
  • 1
    In the MNIST TensorFlow tutorial they define it in terms of one-hot vectors as well. – Lenar Hoyt Jun 22 '16 at 9:32
  • @LenarHoyt When $N=1$, $k_i/N$ would be equivalent to one-hot. You can think of one-hot as the encoding of one item based on its empirical (real) categorical probability. – THN Jul 13 '17 at 11:02
  • 'independent event requires...to encode it' - could you explain this bit please? – Alex Aug 20 '17 at 13:25
  • @Alex This may need longer explanation to understand properly - read up on Shannon-Fano codes and relation of optimal coding to the Shannon entropy equation. To dumb things down, if an event has probability 1/2, your best bet is to code it using a single bit. If it has probability 1/4, you should spend 2 bits to encode it, etc. In general, if your set of events has probabilities of the form 1/2^k, you should give them lengths k - this way your code will approach the Shannon optimal length. – KT. Aug 21 '17 at 9:55

The first logloss formula you are using is for multiclass log loss, where the $i$ subscript enumerates the different classes in an example. The formula assumes that a single $y_i'$ in each example is 1, and the rest are all 0.

That means the formula only captures error on the target class. It discards any notion of errors that you might consider "false positive" and does not care how predicted probabilities are distributed other than predicted probability of the true class.

Another assumption is that $\sum_i y_i = 1$ for the predictions of each example. A softmax layer does this automatically - if you use something different you will need to scale the outputs to meet that constraint.

Question 1

Isn't it a problem that the $y_i$ (in $log(y_i)$) could be 0?

Yes that can be a problem, but it is usually not a practical one. A randomly-initialised softmax layer is extremely unlikely to output an exact 0 in any class. But it is possible, so worth allowing for it. First, don't evaluate $log(y_i)$ for any $y_i'=0$, because the negative classes always contribute 0 to the error. Second, in practical code you can limit the value to something like log( max( y_predict, 1e-15 ) ) for numerical stability - in many cases it is not required, but this is sensible defensive programming.

Question 2

I've learned that cross-entropy is defined as $H_{y'}(y) := - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log (1-y_i)})$

This formulation is often used for a network with one output predicting two classes (usually positive class membership for 1 and negative for 0 output). In that case $i$ may only have one value - you can lose the sum over $i$.

If you modify such a network to have two opposing outputs and use softmax plus the first logloss definition, then you can see that in fact it is the same error measurement but folding the error metric for two classes into a single output.

If there is more than one class to predict membership of, and the classes are not exclusive i.e. an example could be any or all of the classes at the same time, then you will need to use this second formulation. For digit recognition that is not the case (a written digit should only have one "true" class)

  • Note there is some ambiguity in the presentation of the second formula - it could in theory assume just one class and $i$ would then enumerate the examples. – Neil Slater Dec 10 '15 at 16:24
  • I'm sorry, I've asked something different than what I wanted to know. I don't see a problem in $\log(y_i) = 0$, but in $y_i = 0$, because of $\log(y_i)$. Could you please adjust your answer to that? – Martin Thoma Dec 17 '15 at 8:47
  • @NeilSlater if the classes were not mutually exclusive, the output vector for each input may contain more than one 1, should we use the second formula? – Media Feb 28 at 13:15
  • @media: In general, yes you should use sigmoid to get separate probs for each class, and the second loss function that scores each class separately. For some problems however, you might be able to get better results using something more sophisticated/complex. – Neil Slater Feb 28 at 14:26
  • 1
    @Media: Not really. You want to be looking at things such as hierarchical classification though . . . – Neil Slater Feb 28 at 15:38

Given $y_{true}$, you want to optimize your machine learning method to get the $y_{predict}$ as close as possible to $y_{true}$.

First question:

Above answer has explained the background of your first formula, the cross entropy defined in information theory.

From a opinion other than information theory:

you can examine yourself that first formula does not have penalty on false-positiveness(truth is false but your model predict that it is right), while the second one has penalty on false-positiveness. Therefore, the choice of first formula or second, will affect your metrics(aka what statistic quantity you would like to use to evaluate your model).

In layman word:

If you want to accept almost all good people to be your friend but willing to accept some bad people become your friend, then use first formula for criterion.

If you want to punish yourself accepting some bad people to be your friend,but at the same time your good-people accepting rate might be lower than the first condition, then use second formula.

While, I guess most of us are critical and would like to choose the second one(so as many ML package assume what is cross entropy).

Second question:

Cross entropy per sample per class: $$-y_{true}\log{(y_{predict})}$$

Cross entropy for whole datasets whole classes: $$\sum_i^n \sum_k^K -y_{true}^{(k)}\log{(y_{predict}^{(k)})}$$

Thus, when there are only two classes (K = 2), you will have the second formula.

Those issues are handled by the tutorial's use of softmax.

For 1) you're correct that softmax guarantees a non-zero output because it exponentiates it's input. For activations that do not give this guarantee (like relu), it's simple to add a very small positive term to every output to avoid that problem.

As for 2), they aren't the same obviously, but I the softmax formulation they gave takes care of the the issue. If you didn't use softmax, this would cause you to learn huge bias terms that guess 1 for every class for any input. But since they normalize the softmax across all classes, the only way to maximize the output of the correct class is for it to be large relative to the incorrect classes.

  • "you're correct that softmax guarantees a non-zero output" - I know that this is theoretically the case. In reality, can it happen that (due to numeric issues) this becomes 0? – Martin Thoma Dec 10 '15 at 14:30
  • Good question. I assume it's perfectly possible for the exponentiation function to output 0.0 if your input is too small for the precision of your float. However I'd guess most implementations do add the tiny positive term to guarantee non-zero input. – jamesmf Dec 10 '15 at 14:50

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.