3
$\begingroup$

I have a CNN outputting probabilities using a logistic output. The performances are good on the test set. Yet the probabilities it outputs are very clustered, they are either 0 or 1!

enter image description here

I would like some more granularity in the probabilities outputed by the network. Some features are more obvious than other and the network should be able to learn this. Any idea on how I could solve this problem?

$\endgroup$
5
  • 2
    $\begingroup$ This is usually the result you want, provided you have not over-fit. What is the accuracy like in test? Is there some underlying reason why you want less certain probabilities for some examples? If they are inherently uncertain, then you will probably need to label them as such before training (not all NN frameworks can cope with that in the training data, as it removes some optimisation possibilities for softmax). $\endgroup$ Mar 15, 2017 at 12:19
  • $\begingroup$ The accuracy in test is of about 90%. This is a classifier for sentences, and the idea of using a convolution here comes from the fact that features are at the ngram level. You can have ngrams that are positively correlated to the class 1 and others that are negatively correlated to class 1. One sentence can contain both positive and negative features at the same time, this is why I would expect a better calibrated confidence. Here I feel the winning class is the one shouting the loudest. Maybe you are right and I should have labeled my data with a score instead of a binary value. $\endgroup$ Mar 15, 2017 at 13:20
  • 1
    $\begingroup$ The only "calibration" you have is your training and test data. If it is labeled with 0 or 1 for each class, including for those difficult sentences, then that is what your classifier is going to learn. In this case it seems to have learned it well - I don't see a way to get meaningful different probabilities out of this, The existing values are probabilities based on your training data. Yes you could re-scale the output, but it would not correspond to anything statistically from your data. $\endgroup$ Mar 15, 2017 at 14:25
  • $\begingroup$ Ok, but compared to let's say digit recognition, what is the difference? For instance, training on the MNIST dataset with logistic regression would give me this middleground no ? Even though the data has binary labels. $\endgroup$ Mar 15, 2017 at 14:41
  • 1
    $\begingroup$ The difference is in the training data, and how easily/well the model differentiates between classes. In MNIST's case there can be some truly ambiguous examples (such as a '1' that looks very like a '7'), perhaps in your case that is not true, and it is very easy for the network to decide a complex sentence the same way as your labels? I have noticed though, that with your data, you could get 90% accuracy and have relatively high false positive and false negative rates. So it might be that your model actually has a problem such as being overfit. Cannot really tell from what you have shown. $\endgroup$ Mar 15, 2017 at 14:46

1 Answer 1

1
$\begingroup$

That's usually the most ideal case that you get in classification, but if you really want to, you can add a penalty on the predictions to force them to be away from zero or one, e.g. loss'(x) = loss(x) - param * prob_hat(x) * (1 - prob_hat(x)), or some different penalty.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.