Assume that we have two classifiers (C1, C2) and two classes (A, B). These classifiers give us the belonging probability for each class for each instance. Suppose that we have an instance X which is actually should be classified as A. C1 classification result is (1, 0) and C2 classification result is (0.9, 0.1) which means they both classified X correctly as A. Obviously C1 is more confident. Is there any measure that I can use to compare my classifiers based on that?
$-log \space P(y_t|y_p) = -(y_t \space log(y_p) + (1-y_t)\space log(1-y_p))$
Using this metric, confident correct classifications are rewarded more than relatively less confident correct classifications, and confident misclassifications are heavily punished.
Any proper scoring rule meets this criteria when applied to a classification problem.
Some others examples are:
- Brier score (as mentioned by darXider in his comment)
- Spherical Scoring Rule
- Logarithmic Scoring Rule
And an infinite number of other, related functions.