I am working on a multi class classification model where few of the class are with less data compare to other classes. I used random sampling technique to create a sample from the population keeping the proportion of each class equal to that of population. For example, class A has 400 records in the population and class B has 100 records in the population then when doing random sampling I am creating a sample where records of class A and class B are in proportion of 4:1. The trend I have observed is by changing the sample size (keeping inter class proportion constant) of one class leads to change in model performance (accuracy,precision,recall).
What technique do i need to apply in order to make my model stable irrespective of sample size?