Cross-entropy tends to allow errors to change weights even when nodes saturate (which means that their derivatives are asymptotically close to 0.) Link

Why is the above statement true? Figures and examples if possible.

  • 2
    $\begingroup$ Cross-entropy has nothing to do with neural networks per se (the extract talks about activation functions and nodes). The practical reason to use it is that it is a classification loss, and you might have a classification task. It's basically the divergence between the empirical distribution and the prediction distribution. $\endgroup$
    – Emre
    Commented Jun 4, 2017 at 1:39
  • $\begingroup$ @Emre it mentions "network" and "weights". I was confused if you mean something else. $\endgroup$ Commented Jun 4, 2017 at 1:41
  • $\begingroup$ Exactly, this page is misleading you into thinking cross-entropy is intrinsically related to neural networks and that it's an "alternative" to the MSE. Only in the sense that classification is an "alternative" to regression. Just forget you read it and get a real book like AIMA, or PRML $\endgroup$
    – Emre
    Commented Jun 4, 2017 at 1:42
  • $\begingroup$ I understand what cross-entropy is. My question is in the context of neural networks. I don't get what you mean by "classification is an "alternative" to regression". $\endgroup$ Commented Jun 4, 2017 at 1:45
  • $\begingroup$ You asked the practical reason to use cross entropy and I gave it. If you'd really read those books you'd know that. It has nothing to do with saturation in neural networks. $\endgroup$
    – Emre
    Commented Jun 4, 2017 at 2:01

1 Answer 1

  • During back-propagation training, you want to drive output node values to either 1.0 or 0.0 depending on the target values.
  • If you use MSE, the weight adjustment factor (the gradient) contains a term of (output) * (1 – output). As the computed output gets closer and closer to either 0.0 or 1.0 the value of (output) * (1 – output) gets smaller and smaller.
  • For example, if output = 0.6 then (output) * (1 – output) = 0.24 but if output is 0.95 then (output) * (1 – output) = 0.0475.
  • As the adjustment factor gets smaller and smaller, the change in weights gets smaller and smaller and training can stall out, so to speak.
  • But if you use cross-entropy error, the (output) * (1 – output) term goes away (the math is very cool). So, the weight changes don’t get smaller and smaller and so training isn’t s likely to stall out.

Note that this argument assumes you’re doing neural network classification, with either softmax output node activation plus multiclass logloss, or sigmoid output node activation plus binary logloss.

Reference: https://jamesmccaffrey.wordpress.com/2013/11/05/why-you-should-use-cross-entropy-error-instead-of-classification-error-or-mean-squared-error-for-neural-network-classifier-training/


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.