0
$\begingroup$

I'm building a CNN to make a binary classification (1 or zero). For this, I'm using the cost function sigmoid_cross_entropy_with_logits.

But for some reason, the cost using this function is never equal to zero even if the prediction is equal to the correct valuel.

I tried plotting the output using the formula on TensorFlow's website: https://www.tensorflow.org/api_docs/python/tf/nn/sigmoid_cross_entropy_with_logits

This formula:

max(x, 0) - x * z + log(1 + exp(-abs(x)))

And by making this plot, I realized that it really isn't zero when the outputs are equal. For example, if z = 0 and x = 0, the result of this function is ~0.693.

This isn't really making sense to me. Can someone shed some light on why it isn't zero when the prediction is correct?

$\endgroup$

2 Answers 2

1
$\begingroup$

Here is where you are wrong. Sigmoid crossentropy with logits operate with logits instead of the probability hence the calculated loss is based on the logit before sigmoid function is applied.

Now $x= 0$ here is equivalent to : $$P(Y=1|X;\theta)= \frac{1}{1+e^{-0}} = 0.5$$ For shorter notation I shall denote quantity above as $p$.

Hence for the cross entropy loss for a single sample: $$loss(p)=\mathbb{1}_{y=1}\ln{(p)} + \mathbb{1}_{y=0}\ln{(1-p)}$$ WLOG simply because $P(Y=1|X;\theta)=P(Y=0|X;\theta)=0.5$, We have loss equal to $\ln{0.5}$ which is exactly 0.693 that you get.

$\endgroup$
0
$\begingroup$

For a binary classification (1 or 0), you should use softmax cross-entropy with logits since the output classes are mutually exclusive / discrete.

The function you used is for cases with multiple outputs (multilabel) all of which can be ones (their documentation for example talks of the case where image can contain both elephant and a dog). The calculation is different than mutually exclusive case.

$\endgroup$
1
  • $\begingroup$ Isn't it the other wat around? The sigmoid is for the case with only 2 classes (binary classification) ? $\endgroup$
    – WilsonPena
    Aug 16, 2018 at 19:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.