Which Classification Metrics Are Appropriate For Each Class Distribution Scenario?

Question

Currently, I have a balanced dataset (that I artificially over-sampled to make it balanced). My classes are binary (0 or 1). I'm wondering if "accuracy" is the "best" metric to use in the situation when a dataset has roughly equal class balance?

More broadly though, is there a good "rule-of-thumb" or "best practice" regarding which scoring metric (accuracy, precision, recall, AUC, ...) to use based on the dataset's class label "distribution".

Common scenarios I can think of are:

Binary Classification

• high imbalance of 0 class
• high imbalance of 1 class
• roughly equal class quantities
• last, imagine a 75/25 ratio for either

Multiclass Classification

• Majority of samples are dominated by one class (very common in my experience)
• Classes have roughly equally the same amount
• Many class labels with few samples in each (imagine 1000 samples with 100 classes ranging from 2 - 20 samples in each)

I'm aware of techniques like under-sampling and over-sampling to deal with imbalance by altering the dataset's number of samples, but let's say you didn't in this scenario. Which metrics are best for the above scenarios?

Answer 1

First of all don't change the distribution of your data. Your classifier won't perform good at test time if the real data is not balanced.

If you have approximately equal amount of data for each class and they have same importance, accuracy is a good metric which can help you understand how much your classifier has performed well without regarding to the number of labels, binary or multi-class classificaitons.

Binary classification
There is no difference between 0 or 1 because you yourself choose to assign each class to a label but it is common to set the rare and maybe, in some occasions, the desired output to 1.

In cases which data is distributed badly and it is so much skewed it recommended to use F1 score. For last, imagine a 75/25 ratio for either I personally prefer to use F1 because it is unbalanced.

Also in cases where one class is much important than the other class and maybe the data is skewed, it is better to use recall, although you have to change the cost function to emphasize the importance of one class over the other. As an example, suppose that you want to know whether a patient has a bad illness or not. In such cases you try to predict correctly all of those who has that illness because if you tell that someone has an illness but you're wrong, it will have less harm than saying a patient is healthy. So in these cases recall is so much better than precision.

Multi-Class classification

Majority of samples are dominated by one class (very common in my experience)

In this case first of all, if the data is so much skewed, I suggest you using anomaly detection. But if you have more than few data, you can again use F1 score.

Classes have roughly equally the same amount

In this case you can use accuracy if the classes have the same importance. If they don't, again use accuracy but change the cost function to emphasize on the important class.

Many class labels with few samples in each (imagine 1000 samples with 100 classes ranging from 2 - 20 samples in each)

In this case I don't think you will have a good learning. Anyway, you have to use confusion matrix in this for tracking all classes. Although I guess your model will be over-fitted because of the amount of provided data.

---

Consider that in all cases you can use confusion matrix.

---

Finally I suggest you taking a look at here and here.