# Quantifying binary classification uncertainty

I am designing a binary classifier and would like some ideas for how best to quantify uncertainty in the final model classification. The design process consists of first training the model, then performing ROC analysis, and finally choosing a threshold such that the probability of false alarm is ~0.01. So, when an unknown sample is passed into this classifier, it outputs a number between 0 and 1, and if the number is above the threshold, the sample is classified into class 1 (C1), or into C0 if the number is below the threshold.

My question is about how to design a confidence score around the final classification. Say my threshold is 0.9. So, if the model outputs a score of 0.91, then my sample is classified as C1, but if the output score is 0.89, then the sample is classified as C0. Intuitively, given each of these scores, I should have lower confidence in the labels than if the model output was 0.99 or 0.01, respectively. One possibility is to simply compute a normalized [0,1] distance from the score to the threshold, like: $$|\hat{y} - T|/(1 - T)$$ for predictions above the threshold (i.e., samples labeled as C1), and $$|\hat{y} - T|/T$$ for predictions below the threshold (where $$\hat{y}$$ is the model output and $$T$$ is the threshold). One issue with this is that it's not symmetric. That is, for a threshold of 0.9, a model output of 0.91 has an "certainty score" (i.e. the computation shown above) of 0.1, while a model output of 0.89 has an certainty score of 0.011.

Another possibility that occurs to me is the probability of missed detection or the probability of false alarm for instances where $$\hat{y} (sample classified as C0) or $$\hat{y}>T$$ (sample classified as C1), respectively. So, for a model output of 0.89, I would compute $$FNR = 1- sensitivity$$, while for a model output of 0.91, I would compute $$FPR = 1-specificity$$, and use these as measures of uncertainty in the final classification.

Are there other suggested methods for computing this kind of classifier uncertainty?

• As far as I understand your second approach goes in the right direction but I'm not sure if the details are clear. So what you could do is to determine experimentally the rate of error (either FPR or TPR depending on which side of the threshold) for different intervals of predicted $$\hat{y}$$. For example if $$T=0.9$$ you can calculate the FNR for every interval $$[0.0,0.1]\ [0.1,0.2]\ ...\ [0.8,0.9]$$. This way for any predicted $$\hat{y}$$ you can provide the corresponding error rate.