# Non-categorical loss in Keras

I am training a neural network (arbitrary architecture) and I have a label space that is not one-hot encoded, but continuous. The reason is that for the given problem, it is not possible to assign a single class only, it is more of a probability mapping. So in the end, my targets sum up to 1 again, but they are not 1-hot.

I wonder if I am misunderstanding Keras documentation, but for what I read, there is no Crossentropy implementation for this. There is categorical and sparse_categorical (which seem to do exactly the same, but only expect a different label format). My idea was to wrap every single target index into a binary crossentropy, but this feels wrong and I think there is a better solution. Can you please help me find an appropriate CE loss for my task?

• Anything wrong with MSE or MAE? If you do the usual sigmoid activation function at the end, you compress the output values into the unit interval. – Dave Mar 17 '20 at 14:55
• My output is a softmax because the overall probability is still 1. But I want continuous targets, not 1-hot encoded. I struggled to use MSE or MAE because I usually take them for regression problems – PKlumpp Mar 17 '20 at 14:56
• Probability of what? – Dave Mar 17 '20 at 15:00
• Probability of belonging to each out of 4 classes for example. Suppose I have 4 targets, then I will get 4 posteriors. I want to compare these posteriors to 4 targets, which are NOT 1-hot encoded – PKlumpp Mar 17 '20 at 15:03
• So instead of having a 4-vector with a 1 somewhere are 0s elsewhere, you have four probabilities between 0 and 1, where the sum of those four numbers is 1? And this applies for every observation (image or whatever)? If this is the case, please include this information in the question. – Dave Mar 17 '20 at 15:12

It sounds like you want your model to output a probability distribution which matches a "ground truth" probability distribution. Instead of a cross-entropy loss, you should try Kullback-Leiber divergence (keras.losses.kullback_leibler_divergence).

KL-divergence measures the difference between two probability distributions. Minimizing KL-divergence should cause your predicted distribution to match the actual distribution.

By the way, KL-divergence is not just a workaround for Keras's cross-entropy limitations. It's actually the better loss function for this task. From the wikipedia page on KL-divergence (emphasis added by me):

$$D_{KL}(P \| Q) = H(P, Q) - H(P)$$ where $$H(P, Q)$$ is the cross entropy of P and Q, and $$H(P)$$ is the entropy of P (which is the same as the cross-entropy of P with itself).

The KL divergence $$D_{KL}(P \| Q)$$ can be thought of as something like a measurement of how far the distribution Q is from the distribution P. The cross-entropy $$H(P, Q)$$ is itself such a measurement, but it has the defect that $$H(P, P) = H(P)$$ is not zero, so we subtract $$H(P)$$ to make $$D_{KL}(P \| Q)$$ agree more closely with our notion of distance.