# Should softmax cross entropy with logits always be zero if logits and labels are identical?

Apologies for this basic question. I don't have a deep understanding of this function but I was pretty sure it would be zero if logits and label inputs are identical. I found out my MLP model solves my classification problem pretty well but the cost function is never actually 0 (rather it approaches something like -2.14442e+07 but never is actually 0). I have tested what happens if set logits and labels to the same input and still its the same. I use the tensorflow tf.nn.softmax_cross_entropy_with_logits as cost function. Thank you for your help.

Logits are the pre-transform values in a layer, and are not compared directly to the labels when calculating the cost function. In fact, with softmax layer, setting the matching logit to 1, and the rest to zero isn't even that good, because (using $$z_i$$ as logit $$i$$ and $$a_i$$ as activation $$i$$):

$$a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$$

Substitute in your 0 and 1 values for $$z_i$$ and you will see the $$a$$ values are not that close to 0 and 1 (they would be 0.0853 and 0.2319 respectively for 10 classes). It is these activation values that feed into the loss calculation. So with typical multiclass loss function $$l = -\sum_{\forall i} y_i log(a_i)$$, you would score a loss of 1.4611 for such an example.

To get a very low cost, instead set the logit matching the true label (or 1) to a higher value, e.g. 10, and the logits matching the false value (or zero) to a low value e.g. -10. This would lead to

$$a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$$

Where

$$z_i = \{10\; if\;1\;\; |\;\;-10\;if\;0\}$$

And varies between 10 and -10 for values between 1 and 0.

That should significantly reduce the calculated loss (1.855e-08 for those values in a 10 class prediction). The logit allows us to 'stretch' the values for correct and incorrect predictions.

Getting an actual 0 loss when using softmax or sigmoid outputs and logloss is not technically possible. You might see it due to rounding in computation. However, neither sigmoid nor softmax equals exactly 0 or 1, except in the limit as input values tend to $$\pm \infty$$

• @Community, are you sincerely a robot?! – Media Nov 7 '18 at 20:16