Fractions or probabilities as training labels

Question

This it a problem that has come on my path a few times now and I don't have a satisfying solution yet. The goal is to predict probabilities or fractions based on some $x$ where our training $y$ has these probabilties or fractions and thus is in the domain $[0,1]$ as opposed to $\{0,1\}$. My question is with regards to my loss function. In the case of fractions, if the error between 0.4 and 0.5 and the error between 0.89 and 0.99 is the same I can just use MSE if I want to predict the expected value.

In case of probabilities where we want to approach it similarly as classification problems, where the difference between 0.89 and 0.99 is much bigger than that of 0.4 and 0.5, we want to put this in our loss function. Does cross entropy still work properly if I feed it fractions in $y$?

$\mathcal{L}(y,\hat{y})=-y\log(\hat{y}) - (1-y)\log(1-\hat{y})$

Let's say our $y=0.5$ and our current prediction is $\hat{y}=0.6$ we would get:

$\mathcal{L}(0.5,0.6)=-0.5\log(0.6) - 0.5\log(0.4)$

I don't really see why this would go wrong? The function is still convex. Everywhere it says that the target should be in $\{0, 1\}$ however. Maybe my math is lacking or I'm missing something obvious, why is this a bad idea?

Answer 1

Cross-entropy loss still works with probabilities in $[0,1]$ as well as $\{0,1\}$. Most importantly, $\hat{y} = y$ is still a stationary point (although it will not equal $0$).

It is also the case that the possible improvement in loss (and immediate gradient) is larger for $\hat{y} = 0.99, y = 0.89$ than for $\hat{y} = 0.4, y = 0.5$. If you use a sigmoid output, then the gradient at the logit $\hat{y} - y$ still applies - the larger gradient of the loss function scales inversely to the lower gradient of the sigmoid at that point.

So, in short, yes use binary cross-entropy loss for single-class probabilities, even when they are not strictly in $\{0,1\}$.