Cross-entropy loss explanation

Suppose I build a NN for classification. The last layer is a Dense layer with softmax activation. I have five different classes to classify. Suppose for a single training example, the true label is [1 0 0 0 0] while the predictions be [0.1 0.5 0.1 0.1 0.2]. How would I calculate the cross entropy loss for this example?

The cross entropy formula takes in two distributions, $$p(x)$$, the true distribution, and $$q(x)$$, the estimated distribution, defined over the discrete variable $$x$$ and is given by

$$H(p,q) = -\sum_{\forall x} p(x) \log(q(x))$$

For a neural network, the calculation is independent of the following:

• What kind of layer was used.

• What kind of activation was used - although many activations will not be compatible with the calculation because their outputs are not interpretable as probabilities (i.e., their outputs are negative, greater than 1, or do not sum to 1). Softmax is often used for multiclass classification because it guarantees a well-behaved probability distribution function.

For a neural network, you will usually see the equation written in a form where $$\mathbf{y}$$ is the ground truth vector and $$\mathbf{\hat{y}}$$ (or some other value taken direct from the last layer output) is the estimate. For a single example, it would look like this:

$$L = - \mathbf{y} \cdot \log(\mathbf{\hat{y}})$$

where $$\cdot$$ is the vector dot product.

Your example ground truth $$\mathbf{y}$$ gives all probability to the first value, and the other values are zero, so we can ignore them, and just use the matching term from your estimates $$\mathbf{\hat{y}}$$

$$L = -(1\times log(0.1) + 0 \times \log(0.5) + ...)$$

$$L = - log(0.1) \approx 2.303$$

That means, the loss would be same no matter if the predictions are $$[0.1, 0.5, 0.1, 0.1, 0.2]$$ or $$[0.1, 0.6, 0.1, 0.1, 0.1]$$?

Yes, this is a key feature of multiclass logloss, it rewards/penalises probabilities of correct classes only. The value is independent of how the remaining probability is split between incorrect classes.

You will often see this equation averaged over all examples as a cost function. It is not always strictly adhered to in descriptions, but usually a loss function is lower level and describes how a single instance or component determines an error value, whilst a cost function is higher level and describes how a complete system is evaluated for optimisation. A cost function based on multiclass log loss for data set of size $$N$$ might look like this:

$$J = - \frac{1}{N}\left(\sum_{i=1}^{N} \mathbf{y_i} \cdot \log(\mathbf{\hat{y}_i})\right)$$

Many implementations will require your ground truth values to be one-hot encoded (with a single true class), because that allows for some extra optimisation. However, in principle the cross entropy loss can be calculated - and optimised - when this is not the case.

• Okay. That means, the loss would be same no matter if the predictions are [0.1 0.5 0.1 0.1 0.2] or [0.1 0.6 0.1 0.1 0.1]? – Nain Jul 10 '17 at 14:48
• @Nain: That is correct for your example. The cross-entropy loss does not depend on what the values of incorrect class probabilities are. – Neil Slater Jul 10 '17 at 15:25

The answer from Neil is correct. However I think its important to point out that while the loss does not depend on the distribution between the incorrect classes (only the distribution between the correct class and the rest), the gradient of this loss function does effect the incorrect classes differently depending on how wrong they are. So when you use cross-ent in machine learning you will change weights differently for [0.1 0.5 0.1 0.1 0.2] and [0.1 0.6 0.1 0.1 0.1]. This is because the score of the correct class is normalized by the scores of all the other classes to turn it into a probability.

• Can you elaborate it with a proper example? – Nain Nov 14 '17 at 4:42
• @Lucas Adams, can you give an example please ? – koryakinp Jul 28 '18 at 17:15
• The derivative of EACH y_i (softmax output) w.r.t EACH logit z (or the parameter w itself) depends on EVERY y_i. medium.com/@aerinykim/… – Aaron Oct 9 '18 at 8:55

Let's see how the gradient of the loss behaves... We have the cross-entropy as a loss function, which is given by

$$H(p,q) = -\sum_{i=1}^n p(x_i) \log(q(x_i)) = -(p(x_1)\log(q(x_1)) + \ldots + p(x_n)\log(q(x_n))$$

Going from here.. we would like to know the derivative with respect to some $x_i$: $$\frac{\partial}{\partial x_i} H(p,q) = -\frac{\partial}{\partial x_i} p(x_i)\log(q(x_i)).$$ Since all the other terms are cancelled due to the differentiation. We can take this equation one step further to $$\frac{\partial}{\partial x_i} H(p,q) = -p(x_i)\frac{1}{q(x_i)}\frac{\partial q(x_i)}{\partial x_i}.$$

From this we can see that we are still only penalizing the true classes (for which there is value for $p(x_i)$). Otherwise we just have a gradient of zero.

I do wonder how to software packages deal with a predicted value of 0, while the true value was larger than zero... Since we are dividing by zero in that case.

• I think what you want is to take derivative w.r.t. the parameter, not w.r.t. x_i. – Aaron Oct 9 '18 at 6:17

I disagree with Lucas. The values above are already probabilities. Note that the original post indicated that the values had a softmax activation.

The error is only propagated back on the "hot" class and the probability Q(i) does not change if the probabilities within the other classes shift between each other.

• Lucas is correct. With the architecture described by the OP, then gradient at all the logits (as opposed to outputs) is not zero, because the softmax function connects them all. So the [gradient of the] error at the "hot" class propagates to all output neurons. – Neil Slater May 22 '18 at 7:24
• +1 for Neil and Lucas – Aaron Oct 9 '18 at 6:30

The problem is that the probabilities are coming from a 'complicated' function that incorporates the other outputs into the given value. The outcomes are inter-connected, so this way we are not deriving regarding to the actual outcome, but by all the inputs of the last activation function (softmax), for each and every outcome.

I have found a very nice description at deepnotes.io/softmax-crossentropy where the author shows that the actual derivative is $$p_i - y_i$$.

Other neat description at gombru.github.io/2018/05/23/cross_entropy_loss.

I think that using a simple sigmoid as a last activation layer would lead to the approved answer, but using softmax indicates different answer.

• Welcome to Stack Exchange. However what you wrote does not seem to be an answer of the OP's question about calculating cross-entropy loss. – user12075 Sep 25 '18 at 17:05