Confidence intervals for binary classification probabilities

Question

When evaluating a trained binary classification model we often evaluate the misclassification rates, precision-recall, and AUC.

However, one useful feature of classification algorithms are the probability estimates they give, which support the label predictions made by the model.

These probabilities can be useful for a variety of reasons depending on the use case. When using these probabilities it would be useful to have a confidence interval rather than a single point estimate.

So, how can we estimate a probability confidence interval given that the misclassification error may not always serve as a proxy for the error between the estimated probability and the actual probability (which is often unknown)?

I've considered using brier score but I'm sure there is a better way. Can anyone point me in the right direction or offer your own insight?

For example, If I have classes [C0, C1] and my probabilities for a given instance $(x^{(i)}, y^{(i)})$ are {C0: 80, C1:20} then I will classify this instance as C0. Let's suppose that C0 is the correct class label, at this point the model has done it's job and made the correct classification.

I want to go another step further and use the probabilities {C0:80, C1:20} which could be useful for a variety of reasons.

Let's say C0 and C1 respectively represent a customer keeping and closing their account with a bank.

If we wanted to create an expected value $EV$ of dollars at risk of leaving the bank we could calculate $EV$ as $P(C1) * account \space balance$. This would give us a point estimate of $EV$, which is fine but may not tell the whole story given our uncertainty in the model.

So, how can we provide a lower limit and upper limit of the probability this instance is of class C1 with 95% confidence?

Answer 1

I don't think there is a good way to do this for all models, however for a lot of models it's possible to get a sense of uncertainty (this is the keyword you are looking for) in your predictions. I'll list a few:

Bayesian logistic regression gives a probability distribution over probabilities. MCMC can sample weights from your logistic regression (or more sophisticated) model which in turn predict different probabilities. If the variance in these probabilities are high you are less certain about the predictions, you could empirically take the 5% quantile or something.

With neural networks you could train them with dropout (not a bad idea in general) and then instead of testing without the dropout, you do multiple forward passes per prediction and this way you sample from different models. If the variance is high, again you are more uncertain. Variational Inference is another way to sample networks and sample from these different networks to get a measure of uncertainty.

I don't know from the top of my head but I'm sure you could do something with random forests with the variance between the different end nodes where your features end up, assuming they are not deep, but this is just something I thought of.

Answer 2

I don't think that the notion of confidence intervals exists for classifiers.

However you can measure the model uncertainty by looking at the probabilities returned by your model. Be aware that probability is not uncertainty but if you calibrate your probabilities, you can have an idea of the belief of event realisation from a frequentist point of view.

You can expect the error of prediction to be the same as the predicted probability. Indeed with a calibrated classifier if the probability to belong to a certain class is 0.8, you can have 80% chance that your sample actually belongs to this class.