First of all don't change the distribution of your data. Your classifier won't perform good at test time if the real data is not balanced.
If you have approximately equal amount of data for each class and they have same importance, accuracy is a good metric which can help you understand how much your classifier has performed well without regarding to the number of labels, binary or multi-class classificaitons.
Binary classification
There is no difference between 0 or 1 because you yourself choose to assign each class to a label but it is common to set the rare and maybe, in some occasions, the desired output to 1.
In cases which data is distributed badly and it is so much skewed it recommended to use F1
score. For last, imagine a 75/25 ratio for either I personally prefer to use F1
because it is unbalanced.
Also in cases where one class is much important than the other class and maybe the data is skewed, it is better to use recall
, although you have to change the cost function to emphasize the importance of one class over the other. As an example, suppose that you want to know whether a patient has a bad illness or not. In such cases you try to predict correctly all of those who has that illness because if you tell that someone has an illness but you're wrong, it will have less harm than saying a patient is healthy. So in these cases recall
is so much better than precision
.
Multi-Class classification
Majority of samples are dominated by one class (very common in my experience)
In this case first of all, if the data is so much skewed, I suggest you using anomaly detection. But if you have more than few data, you can again use F1
score.
Classes have roughly equally the same amount
In this case you can use accuracy if the classes have the same importance. If they don't, again use accuracy but change the cost function to emphasize on the important class.
Many class labels with few samples in each (imagine 1000 samples with 100 classes ranging from 2 - 20 samples in each)
In this case I don't think you will have a good learning. Anyway, you have to use confusion matrix in this for tracking all classes. Although I guess your model will be over-fitted because of the amount of provided data.
Consider that in all cases you can use
confusion matrix.
Finally I suggest you taking a look at
here and
here.