Does Sampling size matters in Multi classification Model

Question

I am working on a multi class classification model where few of the class are with less data compare to other classes. I used random sampling technique to create a sample from the population keeping the proportion of each class equal to that of population. For example, class A has 400 records in the population and class B has 100 records in the population then when doing random sampling I am creating a sample where records of class A and class B are in proportion of 4:1. The trend I have observed is by changing the sample size (keeping inter class proportion constant) of one class leads to change in model performance (accuracy,precision,recall).

What technique do i need to apply in order to make my model stable irrespective of sample size?

Answer 1

It is always better to keep sample sizes close each other. The problem you are facing is Imbalanced Classification. There are lots of methods you can apply such as upsampling/downsampling, synthetic data generation (check SMOTE).

Model:

I would first convert the model to binary classification such that:

• model 1 predicts: A or not A
• model2 predicts B or not B in "not A" group

Note: Another option three x or not x classifier for three classes)

Then I would apply those sampling techniques (see above) to models. If you have high number of observations try to use downsampling to make class weights as 50:50, if you cant do that try SMOTE to generate data from minority class.

Metrics:

Note that if you don't have balanced dataset, try not to use accuracy as a performance metric. Consider area under ROC, F1 and precision/recall according to your case.

Hope it helps!

Answer 2

Use precision, recall, f1 in weighted mode. There's dedicated library for resampling. What you talk about is stratification. You don't have to code it by hand. There's a method to obtain stratified splits in sklearn.