2
$\begingroup$

I'm trying to predict a probability with a neural network, but having trouble figuring out which loss function is best. Cross entropy was my first thought, but other resources always talk about it in the context of a binary classification problem where the labels are $\{0, 1\}$, but in my case I have an actual probability as the target. Is one of these options clearly best, or maybe are they all valid with just minor differences around the extreme 0/1 regions?

Assuming $x$ is the output of the final layer of my model.

Cross Entropy:
$target * -log(sigmoid(x)) + (1 - target) * -log(1 - sigmoid(x))$

Mean Squared Error with Sigmoid:
$(sigmoid(x) - target)^2$

Mean Squared Error with Clamp:
$(x - target)^2$

When I use the output I clamp the values between $[0, 1]$.

|improve this question
$\endgroup$
  • 1
    $\begingroup$ Cross entropy has been used in logistic regression for decades. Most applications of logistic regression are interested in the predicted probabilities, not developing decision procedures. So I think you're safe to go with cross-entropy. $\endgroup$ – Matthew Drury Feb 9 '19 at 6:45
  • $\begingroup$ Is your target a single scalar that represents the probability or an array with each element represents the probabilities for each class? $\endgroup$ – Louis T Feb 11 '19 at 6:11
  • $\begingroup$ For example, if the goal is to predict the probability of an image contains a cat then your target will be a scalar. Alternatively, you could have a multiclass problem, in this case, we might be interested in predict the probility of an image contains a cat or a dog or a human. Then your target might be a array of probability (i.e [.1, .2., 7] each represents the probability for each class (notice the array adds up to 1). $\endgroup$ – Louis T Feb 11 '19 at 6:15
  • $\begingroup$ @mathew-drury: do you know of any resources talking about the non-classification case? For example it's weird to me that the cross entropy loss when predicting .7 when the target is .7 (ie a perfect prediction) is .61. I mean, the slope at that value is 0 which maybe is all that matters. $\endgroup$ – ahbutfore Feb 11 '19 at 20:37
  • $\begingroup$ @LouisT: The target is a single scalar probability, not really a class though. For example you want a model that predicts the odds of drawing a matching card from a deck given a set of rules as inputs. So if you have the rule "the card must be a heart", the target would be .25. $\endgroup$ – ahbutfore Feb 11 '19 at 20:37
0
$\begingroup$

Given that you are trying to predict a scalar probability value, the cross-entropy formula you listed in the question is only valid if the target variable is discrete. So if your question is "predicts the odds of drawing a matching card from a deck", it will be fine.

The main difference between cross entropy and MSE will be how they penalize wrong predictions. Let's say given a target of 1 but prediction with 0. The cross-entropy is actually undefined in this case, but as the prediction gets closer to 0, the cross-entropy loss gets exponentially larger. On the other hand, your MSE is only 1. Which one is better, it depends on your application, if you want to avoid large margin of errors, it would seem the cross-entropy is more appropriate and vice versa.

The second and third approach only differs in how they make sure the prediction is within [0, 1], one uses a sigmoid function and another uses a clamp. Given you are using a neural network, you should avoid using the clamp function. The clamp function is the same as the identity function within the clamped range, but completely flat outside of it. So the gradient of that function is 1 within the clamp range but 0 outside of it. Because of this, you are more likely to run into the dead neural problem in the same way when people talk about "dead relu".

|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.