I run into this problem from time to time and have always felt like there should be an obvious answer.
I have probabilities for potential classes (from some classifier). I will offer the prediction of the class with the highest probability, however, I would also like to attach a confidence for that prediction.
Example: If I have Classes [C1, C2, C3, C4, C5]
and my Probabilities are {C1: 50, C2: 12, C3: 13, C4: 12, C5:13}
my confidence in predicting C1 should be higher than if I had Probabilities {C1: 50, C2: 45, C3: 2, C4: 1, C5: 2}
.
Reporting that I predict class C1 with 60% probability isn't the whole story. I should be able to derive a confidence from the distribution of probabilities as well. I am certain there is a known method for solving this but I do not know what it is.
EDIT: Taking this to the extreme for clarification: If I had a class C1 with 100% probability (and assuming the classifier had an accurate representation of each class) then I would be extremely confident that C1 was the correct classification. On the other hand if all 5 classes had almost equal probability (Say they are all roughly 20%) than I would be very uncertain claiming that any one was the correct classification. These two extreme cases are more obvious, the challenge is derive a confidence for intermediate examples like the one above.
Any suggestions or references would be of great help.
Thanks in advance.