I was looking at keras source here which calculates cross entropy loss using:

output /= tf.reduce_sum(output,
                        reduction_indices=len(output.get_shape()) - 1,
# manual computation of crossentropy
epsilon = _to_tensor(_EPSILON, output.dtype.base_dtype)
output = tf.clip_by_value(output, epsilon, 1. - epsilon)
return - tf.reduce_sum(target * tf.log(output),
                       reduction_indices=len(output.get_shape()) - 1)

target is the truth data, which is 0 or 1, and output is the output of the neural net.

So it looks like the loss is of the form

$$J_{y'} (y) = - \sum_{i} y_{i}' \log (y_i)$$

where $y_i$ is the model output for class $i$, and $y_i'$ is the truth data.

Does this mean the errors for $y_i' = 0$ do not contribute to the loss? Why isn't the formula

$$J_{y'}(y) = - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log (1-y_i)})$$



1 Answer 1


Does this mean the errors for $y_i=0$ do not contribute to the loss?

That is correct.

However, the respective weights that connect to wrong neurons will still have gradients due to the error, and those gradients will be influenced by the size of each incorrect classification. That is due to how softmax works:

$$\hat{y}_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$$

(where $z_i$ is the pre-softmax value of each neuron, a.k.a. the logit) . . . weights that affect one neuron's pre-transform value affect the post-transform value of all neurons. So those weights will still be adjusted to produce a lower $z_j$ value for the incorrect neurons during weight updates.

Why isn't the formula

$$J_{y'}(y) = - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log (1-y_i)})$$


It is not clear why when selecting a single class, that you would care how probability estimates were distributed amongst incorrect classes, or what the benefit would be to drive the incorrect values to be equal. For instance if $y' = [1, 0, 0, 0]$ then using the suggested formula for $J_{y'}(y)$ gives ~ 0.67 for $y = [0.7, 0.1, 0.1, 0.1]$ and ~0.72 for $y = [0.73, 0.26, 0.05, 0.05]$, yet arguably the second result is better.

However, you would use this loss when dealing with non-exclusive classes (where the outputs would use sigmoid as opposed to softmax activation).

  • $\begingroup$ Thanks, I realize the missing link in my logic was that I was using the softmax activation, where the sum of the probabilities adds up to 1, and so each wrong classification takes away from the "strength" of the correct classification. $\endgroup$ Commented Dec 23, 2017 at 6:25
  • $\begingroup$ @NeilSlater actually I always had this problem but did not know how to ask. Suppose that you have just one sample and the label for that should be zero but the output of softmax is non-zero. Does it ever get updated? the lost is always zero. $\endgroup$ Commented Dec 23, 2017 at 7:47
  • $\begingroup$ based on your answer here and as I know, the second formula should be used for binary tasks to avoid the problem I mentioned. $\endgroup$ Commented Dec 23, 2017 at 8:00
  • 1
    $\begingroup$ @Media: Yes it will get updated, because the softmax transform links all the neuron values together, so even though the loss is effectively only calculated on one neuron's output after the softmax is applied, you will still calculate a non-zero gradient value for all the neurons at the pre-softmax linear (logit) stage using back propagation. $\endgroup$ Commented Dec 23, 2017 at 9:19
  • $\begingroup$ @NeilSlater actually I was thinking about something. In the case that I mentioned, the error is zero which is the least error value that exist although there is 100% error in the typical mentioned situation, back propagation is leading us where? zero error rate isn't optimal? If gradient descent tries to move us to better place, Is there other places better than zero error? because this cost function can not be negative. $\endgroup$ Commented Dec 23, 2017 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.