I'm training a neural network that, for each of six classes, tries to predict the probability that a sample belongs to it. After that, I want to use these probabilities as fractions of the sample belonging to that class. My network gives softmax output and is trained using cross-entropy costs (in fact linear output which is then transformed to softmax by tf.nn.softmax_cross_entropy_with_logits)
Is this the right cost function to use when I want to train the network in getting all 6 probabilities correct instead of just classifying each sample as one of the 6 classes? I started hesitating because the tensorflow documentation says about tf.nn.softmax_cross_entropy_with_logits:
Measures the probability error in discrete classification tasks in which the classes are mutually exclusive (each entry is in exactly one class).
UPDATE Even though it sound strange, the cross-entropy cost function seems to work best here. In my understanding this is because the cross-entropy for discrete probability distributions (wikipedia) is
$H(p, q) = -\sum_x p(x)\, \log q(x).$
This function is minimal when $p(x) = q(x)$ for all $x$. This explains why minimizing the cross-entropy error forces the output distribution $q$ to the target distribution $p$, even though I'm abusing tf.nn.softmax_cross_entropy_with_logits by instead of feeding it labels I feed it a discrete probability distribution.
Why it works better than MSE probably is because the magnitudes of the 6 class probabilities are very different. Class 1 may have a probability of 80% while class 2 has a probability of 0.5%, so MSE pays more attention to getting the 80% class right.
Does this mean that CE is the best option still, or is there a way to scale the outputs or weigh them such that the network pays attention to getting each class correct?