1
$\begingroup$

Assume that we have two classifiers (C1, C2) and two classes (A, B). These classifiers give us the belonging probability for each class for each instance. Suppose that we have an instance X which is actually should be classified as A. C1 classification result is (1, 0) and C2 classification result is (0.9, 0.1) which means they both classified X correctly as A. Obviously C1 is more confident. Is there any measure that I can use to compare my classifiers based on that?

$\endgroup$
2
$\begingroup$

There are many measures which implicitly take into account the confidence of a prediction. One very common one is Log Loss (also called Cross Entropy).

$-log \space P(y_t|y_p) = -(y_t \space log(y_p) + (1-y_t)\space log(1-y_p))$

Using this metric, confident correct classifications are rewarded more than relatively less confident correct classifications, and confident misclassifications are heavily punished.

Any proper scoring rule meets this criteria when applied to a classification problem.

Some others examples are:

And an infinite number of other, related functions.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Could you please mention other measures that you have in your mind? $\endgroup$ – Iman Aug 23 '17 at 19:01
  • $\begingroup$ @Iman Brier score. $\endgroup$ – darXider Aug 23 '17 at 20:35
0
$\begingroup$

In addition to comparing the log probabilities as Thomas had suggested, you can run cross validation and compare the means and standard deviations of your models' errors. Then basic statistics can be used to compare these distributions.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.