I am working on a multi class classification model where few of the class are with less data compare to other classes. I used random sampling technique to create a sample from the population keeping the proportion of each class equal to that of population. For example, class A has 400 records in the population and class B has 100 records in the population then when doing random sampling I am creating a sample where records of class A and class B are in proportion of 4:1. The trend I have observed is by changing the sample size (keeping inter class proportion constant) of one class leads to change in model performance (accuracy,precision,recall).

What technique do i need to apply in order to make my model stable irrespective of sample size?


It is always better to keep sample sizes close each other. The problem you are facing is Imbalanced Classification. There are lots of methods you can apply such as upsampling/downsampling, synthetic data generation (check SMOTE).


I would first convert the model to binary classification such that:

  • model 1 predicts: A or not A
  • model2 predicts B or not B in "not A" group

Note: Another option three x or not x classifier for three classes)

Then I would apply those sampling techniques (see above) to models. If you have high number of observations try to use downsampling to make class weights as 50:50, if you cant do that try SMOTE to generate data from minority class.


Note that if you don't have balanced dataset, try not to use accuracy as a performance metric. Consider area under ROC, F1 and precision/recall according to your case.

Hope it helps!


Use precision, recall, f1 in weighted mode. There's dedicated library for resampling. What you talk about is stratification. You don't have to code it by hand. There's a method to obtain stratified splits in sklearn.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.