Loss Function for Probability Regression

Question

I am trying to predict a probability with a neural network, but having trouble figuring out which loss function is best. Cross entropy was my first thought, but other resources always talk about it in the context of a binary classification problem where the labels are $\{0, 1\}$, but in my case I have an actual probability as the target. Is one of these options clearly best, or maybe are they all valid with just minor differences around the extreme 0/1 regions?

Assuming $x$ is the output of the final layer of my model.

Cross Entropy:
$\text{target} * -\log(\text{sigmoid}(x)) + (1 - \text{target}) * -\log(1 - \text{sigmoid}(x))$

Mean Squared Error with Sigmoid:
$(\text{sigmoid}(x) - \text{target})^2$

Mean Squared Error with Clamp:
$(x - \text{target})^2$

When I use the output I clamp the values between $[0, 1]$.

Answer 1

Given that you are trying to predict a scalar probability value, the cross-entropy formula you listed in the question is only valid if the target variable is discrete. So if your question is "predicts the odds of drawing a matching card from a deck", it will be fine.

The main difference between cross entropy and MSE will be how they penalize wrong predictions. Let's say given a target of 1 but prediction with 0. The cross-entropy is actually undefined in this case, but as the prediction gets closer to 0, the cross-entropy loss gets exponentially larger. On the other hand, your MSE is only 1. Which one is better, it depends on your application, if you want to avoid large margin of errors, it would seem the cross-entropy is more appropriate and vice versa.

The second and third approach only differs in how they make sure the prediction is within [0, 1], one uses a sigmoid function and another uses a clamp. Given you are using a neural network, you should avoid using the clamp function. The clamp function is the same as the identity function within the clamped range, but completely flat outside of it. So the gradient of that function is 1 within the clamp range but 0 outside of it. Because of this, you are more likely to run into the dead neural problem in the same way when people talk about "dead relu".

Answer 2

In the case when the domain of the output variable is discrete or continuous, and not binary, you could also consider the KL-divergence as the loss to compute the distance between the predicted probability distribution and the target one, for example.

This is important also because in some implementations, for efficiency reasons, the cross entropy only get indexes as target variables and not distributions. This is the case with pytorch, e.g. see this.

Answer 3

If the target is probability/rates (observations limited to the open interval (0, 1)), beta regression is a useful model.

It is covered by Ferrari and Cribari-Neto in their paper "Beta regression for modeling rates and proportions" (2004).

Answer 4

In practice, cross entropy is better than mean squared error with sigmoid because cross entropy loss is convex w.r.t the last layer whereas MSE with sigmoid is not. This means that it may get stuck at local minimum and as a result the learned probability may never reach the target value.

On the other hand, it's non-trivial to argue if cross entropy is distinctly better than MSE with clamp. They actually produce the same gradient values. You can refer to the discussion on Using MSE instead of log-loss in logistic regression, where the conclusion seems to be that the two are computationally similar but from a probabilistic view, cross entropy is more intuitive and provides more useful results than just the output. I'd also argue that with a neural network model, the output will be too unstable during extrapolation that a simple clamping into [0,1] would not produce helpful result.

Regarding your concern that

It's weird to me that the cross entropy loss when predicting .7 when the target is .7 (ie a perfect prediction) is .61

The value of loss function doesn't always have practical meaning. Especially in probabilistic contexts, the value of metrics such as cross entropy and KL-divergence are mostly relative than absolute, meaning that they only makes sense when used in comparison, not much when used standalone. What really matters is the gradient value induced by them, which has a decisive role in optimization and model learning.