1
$\begingroup$

I'm training a neural network that, for each of six classes, tries to predict the probability that a sample belongs to it. After that, I want to use these probabilities as fractions of the sample belonging to that class. My network gives softmax output and is trained using cross-entropy costs (in fact linear output which is then transformed to softmax by tf.nn.softmax_cross_entropy_with_logits)

Is this the right cost function to use when I want to train the network in getting all 6 probabilities correct instead of just classifying each sample as one of the 6 classes? I started hesitating because the tensorflow documentation says about tf.nn.softmax_cross_entropy_with_logits:

Measures the probability error in discrete classification tasks in which the classes are mutually exclusive (each entry is in exactly one class).

UPDATE Even though it sound strange, the cross-entropy cost function seems to work best here. In my understanding this is because the cross-entropy for discrete probability distributions (wikipedia) is

$H(p, q) = -\sum_x p(x)\, \log q(x).$

This function is minimal when $p(x) = q(x)$ for all $x$. This explains why minimizing the cross-entropy error forces the output distribution $q$ to the target distribution $p$, even though I'm abusing tf.nn.softmax_cross_entropy_with_logits by instead of feeding it labels I feed it a discrete probability distribution.

Why it works better than MSE probably is because the magnitudes of the 6 class probabilities are very different. Class 1 may have a probability of 80% while class 2 has a probability of 0.5%, so MSE pays more attention to getting the 80% class right.

Does this mean that CE is the best option still, or is there a way to scale the outputs or weigh them such that the network pays attention to getting each class correct?

Improve this question
$\endgroup$
6
  • $\begingroup$ It's vague, can a sample belong to multiple classes or only one? $\endgroup$
    – Thomas Wagenaar
    Jun 12, 2017 at 14:59
  • $\begingroup$ Sorry if I was not clear! It can belong to multiple classes since I interpret the class probability as the fraction of the sample belonging to that class (e.g. if my sample is 100 people, then 0.2 output of neuron 1 will be interpreted as 20 people in class 1) $\endgroup$
    – Robo
    Jun 12, 2017 at 15:20
  • $\begingroup$ So the 100 people always get divided among the classes? And all of them certainly belong to a certain class? $\endgroup$
    – Thomas Wagenaar
    Jun 12, 2017 at 15:21
  • $\begingroup$ Correct! They are grouped in 1 sample because they have the same characteristics $\endgroup$
    – Robo
    Jun 12, 2017 at 15:22
  • $\begingroup$ stats.stackexchange.com/questions/207794/… interesting $\endgroup$
    – Thomas Wagenaar
    Jun 12, 2017 at 16:02

1 Answer 1

Reset to default
1
$\begingroup$

Cross entropy is indeed appropriate for multiclass classification.

When the tensorflow documentation states about cross_entropy that:

"Measures the probability error in discrete classification tasks in which the classes are mutually exclusive (each entry is in exactly one class)."

it is referring to the fact that, at a conceptual level, classes are not expected to overlap. An example of non-overlapping classes are "cat" and "dog". An example of overlapping classes are "Elephant" and "African Elephant"; this type of problem is called multilabel classification, because instead of classes, you have labels and each individual can be assigned many labels.

Update: with the new information about the problem, we can tell that the problem being faced is not a classification one, as the desired output are specific probability values. This implies that the problem is a regression one. As the outputs are probabilities, it is appropriate to use softmax to ensure they add up to 1. As it is regression, it would be appropriate to use MSE as loss function.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you for your answer! Does it also count as multiclass classification if 1 sample is 20% class 1, 30% class 2 etc. and these percentages matter? $\endgroup$
    – Robo
    Jun 12, 2017 at 15:39
  • $\begingroup$ Please confirm this is what you mean: "You have a piece of input data $x$ and the expected output (which is part of the training data) is a list of percentages, which are the degree in which $x$ belongs to some classes; this way, when given $x$, the model would output percentages that are as close as possible to those supplied as training data". If so, then you don't have a classification problem, but a regression one, in which you want to predict the specific values of the percentages. $\endgroup$
    – noe
    Jun 12, 2017 at 16:57
  • $\begingroup$ That is indeed what I am trying to do. But this means I do want my outputs to be probabilities and add up to 1. Does that mean I should use softmax output but a different cost function like MSE? $\endgroup$
    – Robo
    Jun 13, 2017 at 7:56
  • $\begingroup$ Yes, exactly that: softmax + MSE. $\endgroup$
    – noe
    Jun 13, 2017 at 8:16
  • $\begingroup$ Strangely enough, learning to minimize CE cost does give better results (when optimizing the CE, the MSE of the results is even 2 times lower than when minimizing the MSE itself). To me that sounds very strange, since the CE is not even a proper cost function if the targets are not in {0,1}. It is unclear what exactly it is learning now, but it learns something apparently $\endgroup$
    – Robo
    Jun 13, 2017 at 9:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.