1
$\begingroup$

I am training a neural network (arbitrary architecture) and I have a label space that is not one-hot encoded, but continuous. The reason is that for the given problem, it is not possible to assign a single class only, it is more of a probability mapping. So in the end, my targets sum up to 1 again, but they are not 1-hot.

I wonder if I am misunderstanding Keras documentation, but for what I read, there is no Crossentropy implementation for this. There is categorical and sparse_categorical (which seem to do exactly the same, but only expect a different label format). My idea was to wrap every single target index into a binary crossentropy, but this feels wrong and I think there is a better solution. Can you please help me find an appropriate CE loss for my task?

Improve this question
$\endgroup$
6
  • $\begingroup$ Anything wrong with MSE or MAE? If you do the usual sigmoid activation function at the end, you compress the output values into the unit interval. $\endgroup$
    – Dave
    Commented Mar 17, 2020 at 14:55
  • $\begingroup$ My output is a softmax because the overall probability is still 1. But I want continuous targets, not 1-hot encoded. I struggled to use MSE or MAE because I usually take them for regression problems $\endgroup$
    – PKlumpp
    Commented Mar 17, 2020 at 14:56
  • $\begingroup$ Probability of what? $\endgroup$
    – Dave
    Commented Mar 17, 2020 at 15:00
  • $\begingroup$ Probability of belonging to each out of 4 classes for example. Suppose I have 4 targets, then I will get 4 posteriors. I want to compare these posteriors to 4 targets, which are NOT 1-hot encoded $\endgroup$
    – PKlumpp
    Commented Mar 17, 2020 at 15:03
  • $\begingroup$ So instead of having a 4-vector with a 1 somewhere are 0s elsewhere, you have four probabilities between 0 and 1, where the sum of those four numbers is 1? And this applies for every observation (image or whatever)? If this is the case, please include this information in the question. $\endgroup$
    – Dave
    Commented Mar 17, 2020 at 15:12

2 Answers 2

Reset to default
1
$\begingroup$

It sounds like you want your model to output a probability distribution which matches a "ground truth" probability distribution. Instead of a cross-entropy loss, you should try Kullback-Leiber divergence (keras.losses.kullback_leibler_divergence).

KL-divergence measures the difference between two probability distributions. Minimizing KL-divergence should cause your predicted distribution to match the actual distribution.

By the way, KL-divergence is not just a workaround for Keras's cross-entropy limitations. It's actually the better loss function for this task. From the wikipedia page on KL-divergence (emphasis added by me):

$$D_{KL}(P \| Q) = H(P, Q) - H(P)$$ where $H(P, Q)$ is the cross entropy of P and Q, and $H(P)$ is the entropy of P (which is the same as the cross-entropy of P with itself).

The KL divergence $D_{KL}(P \| Q)$ can be thought of as something like a measurement of how far the distribution Q is from the distribution P. The cross-entropy $H(P, Q)$ is itself such a measurement, but it has the defect that $H(P, P) = H(P)$ is not zero, so we subtract $H(P)$ to make $D_{KL}(P \| Q)$ agree more closely with our notion of distance.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I was also thinking about this. Good Point. For completeness, I will also add another answer to this question about categorical crossentropy $\endgroup$
    – PKlumpp
    Commented Mar 17, 2020 at 19:03
0
$\begingroup$

While looking at the source of Keras' Categorical Cross-Entropy implementation, I found that it can apply a label smoothing. Therefore, it should also be able to process inputs which are not 1-hot encoded. Testing this with TF 2.1.0 worked for me.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.