Cross-entropy tends to allow errors to change weights even when nodes saturate (which means that their derivatives are asymptotically close to 0.) Link
Why is the above statement true? Figures and examples if possible.
$\begingroup$
$\endgroup$
9
-
2$\begingroup$ Cross-entropy has nothing to do with neural networks per se (the extract talks about activation functions and nodes). The practical reason to use it is that it is a classification loss, and you might have a classification task. It's basically the divergence between the empirical distribution and the prediction distribution. $\endgroup$– EmreCommented Jun 4, 2017 at 1:39
-
$\begingroup$ @Emre it mentions "network" and "weights". I was confused if you mean something else. $\endgroup$– Abhishek BhatiaCommented Jun 4, 2017 at 1:41
-
$\begingroup$ Exactly, this page is misleading you into thinking cross-entropy is intrinsically related to neural networks and that it's an "alternative" to the MSE. Only in the sense that classification is an "alternative" to regression. Just forget you read it and get a real book like AIMA, or PRML $\endgroup$– EmreCommented Jun 4, 2017 at 1:42
-
$\begingroup$ I understand what cross-entropy is. My question is in the context of neural networks. I don't get what you mean by "classification is an "alternative" to regression". $\endgroup$– Abhishek BhatiaCommented Jun 4, 2017 at 1:45
-
$\begingroup$ You asked the practical reason to use cross entropy and I gave it. If you'd really read those books you'd know that. It has nothing to do with saturation in neural networks. $\endgroup$– EmreCommented Jun 4, 2017 at 2:01
|
Show 4 more comments
1 Answer
$\begingroup$
$\endgroup$
- During back-propagation training, you want to drive output node values to either 1.0 or 0.0 depending on the target values.
- If you use MSE, the weight adjustment factor (the gradient) contains a term of (output) * (1 – output). As the computed output gets closer and closer to either 0.0 or 1.0 the value of (output) * (1 – output) gets smaller and smaller.
- For example, if output = 0.6 then (output) * (1 – output) = 0.24 but if output is 0.95 then (output) * (1 – output) = 0.0475.
- As the adjustment factor gets smaller and smaller, the change in weights gets smaller and smaller and training can stall out, so to speak.
- But if you use cross-entropy error, the (output) * (1 – output) term goes away (the math is very cool). So, the weight changes don’t get smaller and smaller and so training isn’t s likely to stall out.
Note that this argument assumes you’re doing neural network classification, with either softmax output node activation plus multiclass logloss, or sigmoid output node activation plus binary logloss.
answered Jun 4, 2017 at 4:50