4
$\begingroup$

The Tensorflow's documentation of softmax_cross_entropy_with_logits:

Measures the probability error in discrete classification tasks in which the classes are mutually exclusive (each entry is in exactly one class). For example, each CIFAR-10 image is labeled with one and only one label: an image can be a dog or a truck, but not both.

NOTE: While the classes are mutually exclusive, their probabilities need not be. All that is required is that each row of labels is a valid probability distribution. If they are not, the computation of the gradient will be incorrect.

At first glance it may seen contradictory, but my guess is that, provided these conditions for the classes and for the labels, the probability distribution of the labels have always one maximum. For instance, labels = [0.5 0.5] is a valid probability distribution in a binary case but it does not comply with the exclusivity of the classes. If I'm wrong, I don't get the idea behind the documentation.

It is also pointed out that for mutually elusive probabilities, sparse_categorical_cross_entropy should be used, but I think that softmax_cross_entropy_with_logits can be used if the labels are one-hot encoded, which is a valid probability distribution (a deterministic one)

Edit:

When going around this question again, I'm wondering if in a multi-class exclusive case where the only constraint on the labels are that they have to be a valid probability distribution, labels = [0.5 0.5] should be a valid instance label. This label means that the annotator nor the net can tell if this ground-truth instance belongs to class_0 or class_1...

Improve this question
$\endgroup$
3
  • $\begingroup$ There are two distributions (ground truth, model distribution), we use the cross-entropy loss function to measure their distance. thus for the backpropagation to work, the probability distribution of the labels doesn't have to be one hot encoded. for the exclusive case maybe they are modeling the noise (difficult examples). $\endgroup$
    – Fadi Bakoura
    Commented May 23, 2018 at 11:42
  • $\begingroup$ Thx, I was thinking in something similar. My conclusion is: although softmax_cross_entropy_with_logits needs labels to be a valid probability distribution (not-only the one-hot encoding) it is very rare that for a mutli-class case the probability distribution of the labels are not of this form. So, as a rule of thumb, use this loss-function with one-hot encoded labels (though not necessary, but rare). $\endgroup$
    – ignatius
    Commented May 23, 2018 at 12:34
  • $\begingroup$ In this link there is a nice discussion about this topic, and it covers the rare case in which for multi-class classification, soft-labels are accepted <stackoverflow.com/questions/47034888/…> $\endgroup$
    – ignatius
    Commented May 23, 2018 at 14:22

1 Answer 1

Reset to default
-2
$\begingroup$

What does it mean that classes are mutually exclusive but soft-labels are accepted?

As it can be seen from here, tf.nn.softmax produces just the result of applying the softmax function to an input tensor. The softmax "squishes" the inputs so that sum(input) = 1; it's a way of normalizing. The shape of the output of a softmax is the same as the input - it just normalizes the values. The outputs of softmax can be interpreted as probabilities. In contrast, tf.nn.softmax_cross_entropy_with_logits computes the cross-entropy of the result after applying the softmax function (but it does it all together in a more mathematically careful way). It's mathematically careful due to the fact that $y_i$ in $log(y_i)$ can be zero. As you can read from here, A randomly-initialized softmax layer is extremely unlikely to output an exact 0 in any class. But it is possible, so worth allowing for it. First, don't evaluate log(yi) for any y′i=0, because the negative classes always contribute 0 to the error. Second, in the practical code you can limit the value to something like $log( max( y_predict, 1e-15 ) )$ for numerical stability - in many cases, it is not required, but this is sensible defensive programming. I encourage you to take a look at the answers to this question.

NOTE: While the classes are mutually exclusive, their probabilities need not be. All that is required is that each row of labels is a valid probability distribution. If they are not, the computation of the gradient will be incorrect.

The first sentence means that your classifier may not be able to classify the labels exactly as they are, one-hot-encoded. What it does is to find the chance that how likely it is that the input belongs to each class. And this won't make a problem if it does not have mutually exclusive output vector as the algorithm implies. It just needs a vector that the sum of its entries is equal to one. If they are not, the computation of the gradient will be incorrect. I guess this line is added to announce that the outputs of this differentiable component will not be one-hot-encoded and it's because of the nature of these nets. The first layers of convolutional networks are like bases vectors and each instance of classes share these bases. All inputs are made up of these bases.

I'm wondering if, in a multi-class exclusive case where the only constraint on the labels is that they have to be a valid probability distribution, labels = [0.5 0.5] should be a valid instance label. This label means that the annotator nor the net can tell if this ground-truth instance belongs to class_0 or class_1...

Basically, for multi-label classification, your input may have different labels. Consequently, your classes won't be mutually exclusive anymore. Moreover, in those cases, we don't use softmax as the last layer. You have to have sigmoid for each output and your cross-entropy cost function will be slightly different 1. Consequently, the output of each entry should be a valid probability, that is why sigmoid is used, and the label vectors for such tasks is a not one-hot-encoded anymore. different classes have different entries and based on the existence of instances of each class, the corresponding entry should be one.

Improve this answer
$\endgroup$
12
  • $\begingroup$ I appreciate your response, but I think your are answering it wrong, at least for the first two paragraphs. I feel you're confusing the labels with the predictions. I understand that for a multi-class classifier, the predictions are probabilities, given by the softmax, for example, But it is not he point, the question is how can soft-labels be accepted in a multi-class exclusive case. $\endgroup$
    – ignatius
    Commented May 23, 2018 at 12:23
  • $\begingroup$ @ignatius While the classes are mutually exclusive, their probabilities need not be. This sentence is for prediction time. Moreover, the labels you provide to your network should be strictly defined. It means they should have a specified label. If you provide probabilities with ambiguity, you can not expect your network to find useful things. $\endgroup$
    – Green Falcon
    Commented May 23, 2018 at 12:31
  • $\begingroup$ I'm not completely agree, sorry... Although in While the classes are mutually exclusive, their probabilities need not be referees to the probabilities of the predictions, in All that is required is that each row of labels is a valid probability distribution does not say that the labels have to be one-hot encoded, but a valid probability distribution. It is confusing for me, can you give an example of a valid probability distribution for the labels that are not one-hot encoded? $\endgroup$
    – ignatius
    Commented May 23, 2018 at 12:37
  • $\begingroup$ It is a valid distribution because you have passed the logits, the linear part of the last layer for each neuron, as the input to the softmax layer. softmax layer is a mapping. $softmax:R^m−>R^m$. What softmax does is normalizing the inputs to find the probability for each output. Consequently, whatever you have in the last layer's logits, the softmax layers turns them to valid distribution, the sum of all outputs will be equal to one. This is also true for mutually exclusive labels. They are one-hot-encoded and they sum to one. $\endgroup$
    – Green Falcon
    Commented May 23, 2018 at 12:44
  • $\begingroup$ I know that, but this is not what we are discussing here... the input tensor labels for softmax_cross_entropy_with_logits are the ground truth annotation of each training instance... Yes, the softmax turns the logits to a valid distribution, but again, logits are logits and labels are labels (ground-truth annotations) $\endgroup$
    – ignatius
    Commented May 23, 2018 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.