Deriving Confidences from Distribution of Class Probabilities for a Prediction

Question

I run into this problem from time to time and have always felt like there should be an obvious answer.

I have probabilities for potential classes (from some classifier). I will offer the prediction of the class with the highest probability, however, I would also like to attach a confidence for that prediction.

Example: If I have Classes [C1, C2, C3, C4, C5] and my Probabilities are {C1: 50, C2: 12, C3: 13, C4: 12, C5:13} my confidence in predicting C1 should be higher than if I had Probabilities {C1: 50, C2: 45, C3: 2, C4: 1, C5: 2}.

Reporting that I predict class C1 with 60% probability isn't the whole story. I should be able to derive a confidence from the distribution of probabilities as well. I am certain there is a known method for solving this but I do not know what it is.

EDIT: Taking this to the extreme for clarification: If I had a class C1 with 100% probability (and assuming the classifier had an accurate representation of each class) then I would be extremely confident that C1 was the correct classification. On the other hand if all 5 classes had almost equal probability (Say they are all roughly 20%) than I would be very uncertain claiming that any one was the correct classification. These two extreme cases are more obvious, the challenge is derive a confidence for intermediate examples like the one above.

Any suggestions or references would be of great help.

Thanks in advance.

Answer 1

As @David says, in your initial example, your confidence about C1 is the same in both cases. In your second example, you most certainly are less confident about the most-probable class in the second case, since the most-probable class is far less probable!

You may have to unpack what you're getting at when you say 'confidence' then, since here you're not using it as a term of art but an English word.

I suspect you may be looking for the idea of entropy, or uncertainty present in the distribution of all class probabilities. In your first example, it is indeed lower in the second case than the first. I don't think what you're getting at is just a function of the most-probable class, that is.

Answer 2

If I have Classes [C1, C2, C3, C4, C5] and my Probabilities are {C1: 50, C2: 12, C3: 13, C4: 12, C5:13} my confidence in predicting C1 should be higher than if I had Probabilities {C1: 50, C2: 45, C3: 2, C4: 1, C5: 2}.

Assuming that those probabilities are accurate, this isn't true. In your second case you can be a lot more confident that the ground truth is one of C1 or C2, but in terms of absolute confidence about C1 the probability is the same across both examples. To illustrate this with a more clear example, if you had a 100 sided die that had 50 sides labeled with "C1" then the labels on the the other 50 sides are irrelevant to the likelihood that you would roll a "C1".

Now with that said, your probabilities from your model are most certainly not perfect so there may be a way to use the intra-class correlations to improve them. Can you provide some more details about your specific problem and modeling workflow that you have used to get your probabilities?