Quantifying binary classification uncertainty

Question

I am designing a binary classifier and would like some ideas for how best to quantify uncertainty in the final model classification. The design process consists of first training the model, then performing ROC analysis, and finally choosing a threshold such that the probability of false alarm is ~0.01. So, when an unknown sample is passed into this classifier, it outputs a number between 0 and 1, and if the number is above the threshold, the sample is classified into class 1 (C1), or into C0 if the number is below the threshold.

My question is about how to design a confidence score around the final classification. Say my threshold is 0.9. So, if the model outputs a score of 0.91, then my sample is classified as C1, but if the output score is 0.89, then the sample is classified as C0. Intuitively, given each of these scores, I should have lower confidence in the labels than if the model output was 0.99 or 0.01, respectively. One possibility is to simply compute a normalized [0,1] distance from the score to the threshold, like: $|\hat{y} - T|/(1 - T)$ for predictions above the threshold (i.e., samples labeled as C1), and $|\hat{y} - T|/T$ for predictions below the threshold (where $\hat{y}$ is the model output and $T$ is the threshold). One issue with this is that it's not symmetric. That is, for a threshold of 0.9, a model output of 0.91 has an "certainty score" (i.e. the computation shown above) of 0.1, while a model output of 0.89 has an certainty score of 0.011.

Another possibility that occurs to me is the probability of missed detection or the probability of false alarm for instances where $\hat{y}<T$ (sample classified as C0) or $\hat{y}>T$ (sample classified as C1), respectively. So, for a model output of 0.89, I would compute $FNR = 1- sensitivity$, while for a model output of 0.91, I would compute $FPR = 1-specificity$, and use these as measures of uncertainty in the final classification.

Are there other suggested methods for computing this kind of classifier uncertainty?

Answer 1

First things first: be sure to use a validation set in order to determine the optimal threshold, do not use the final test set.

Another general point: it depends what you mean by "false alarm", but in general you should not assume that it is possible for the classifier to keep the error rate under a predefined value (unless you have already done an experiment which shows this, of course). If by "false alarm" you mean only one type of error then it is usually possible to keep one type of error under a threshold at the expense of the other type of error. For instance if the goal is to minimize false positive errors, then one can let the classifier make as many false negative errors as necessary in order to keep the false positive rate under the threshold.

Now about quantifying uncertainty:

• The first approach that you propose makes sense, but as you said it's not symmetric and also it wouldn't allow to convey uncertainty in any practical way. Ideally you'd want to be able to say something like "there's an X% chance that the prediction is correct" but you cannot obtain that from a difference with the threshold.
• As far as I understand your second approach goes in the right direction but I'm not sure if the details are clear. So what you could do is to determine experimentally the rate of error (either FPR or TPR depending on which side of the threshold) for different intervals of predicted $\hat{y}$. For example if $T=0.9$ you can calculate the FNR for every interval $[0.0,0.1]\ [0.1,0.2]\ ...\ [0.8,0.9]$. This way for any predicted $\hat{y}$ you can provide the corresponding error rate.