# Loss Function for Probability Regression

I'm trying to predict a probability with a neural network, but having trouble figuring out which loss function is best. Cross entropy was my first thought, but other resources always talk about it in the context of a binary classification problem where the labels are $$\{0, 1\}$$, but in my case I have an actual probability as the target. Is one of these options clearly best, or maybe are they all valid with just minor differences around the extreme 0/1 regions?

Assuming $$x$$ is the output of the final layer of my model.

Cross Entropy:
$$target * -log(sigmoid(x)) + (1 - target) * -log(1 - sigmoid(x))$$

Mean Squared Error with Sigmoid:
$$(sigmoid(x) - target)^2$$

Mean Squared Error with Clamp:
$$(x - target)^2$$

When I use the output I clamp the values between $$[0, 1]$$.

• Cross entropy has been used in logistic regression for decades. Most applications of logistic regression are interested in the predicted probabilities, not developing decision procedures. So I think you're safe to go with cross-entropy. – Matthew Drury Feb 9 '19 at 6:45
• Is your target a single scalar that represents the probability or an array with each element represents the probabilities for each class? – Louis T Feb 11 '19 at 6:11
• For example, if the goal is to predict the probability of an image contains a cat then your target will be a scalar. Alternatively, you could have a multiclass problem, in this case, we might be interested in predict the probility of an image contains a cat or a dog or a human. Then your target might be a array of probability (i.e [.1, .2., 7] each represents the probability for each class (notice the array adds up to 1). – Louis T Feb 11 '19 at 6:15
• @mathew-drury: do you know of any resources talking about the non-classification case? For example it's weird to me that the cross entropy loss when predicting .7 when the target is .7 (ie a perfect prediction) is .61. I mean, the slope at that value is 0 which maybe is all that matters. – ahbutfore Feb 11 '19 at 20:37
• @LouisT: The target is a single scalar probability, not really a class though. For example you want a model that predicts the odds of drawing a matching card from a deck given a set of rules as inputs. So if you have the rule "the card must be a heart", the target would be .25. – ahbutfore Feb 11 '19 at 20:37