Is there any measure to find how much classifiers are confident?

Assume that we have two classifiers (C1, C2) and two classes (A, B). These classifiers give us the belonging probability for each class for each instance. Suppose that we have an instance X which is actually should be classified as A. C1 classification result is (1, 0) and C2 classification result is (0.9, 0.1) which means they both classified X correctly as A. Obviously C1 is more confident. Is there any measure that I can use to compare my classifiers based on that?

There are many measures which implicitly take into account the confidence of a prediction. One very common one is Log Loss (also called Cross Entropy).

$-log \space P(y_t|y_p) = -(y_t \space log(y_p) + (1-y_t)\space log(1-y_p))$

Using this metric, confident correct classifications are rewarded more than relatively less confident correct classifications, and confident misclassifications are heavily punished.

Any proper scoring rule meets this criteria when applied to a classification problem.

Some others examples are:

And an infinite number of other, related functions.

• Could you please mention other measures that you have in your mind?
– Iman
Aug 23, 2017 at 19:01
• @Iman Brier score. Aug 23, 2017 at 20:35

In addition to comparing the log probabilities as Thomas had suggested, you can run cross validation and compare the means and standard deviations of your models' errors. Then basic statistics can be used to compare these distributions.