Imbalanced training set vs smaller balanced training set?

Say I am using a maximum likelihood approach and my output unit computes a softmax function. My training set is distributed as follows over 6 classes:

class_samples[0]=23, class_samples[1]=5, class_samples[2]=44,
class_samples[3]=14, class_samples[4]=19, class_samples[5]=31


What should I do?

1. use the training set as given above with a normalizing weight balancing(e.g. using sklearn.utils.class_weight.compute_class_weight).

2. or should I simply use the minimum number of samples in a class(i.e. 5) to extract a balanced distribution of examples?

Why should I choose one over the other? Intuitively, I would think that using as many training examples as possible is the better option. However, I have tried to do some computations but I fail to show that usage of all examples with a normalizing weight balancing is better.

I have of course tried to do some heavy research but for some reason I cannot find the answer. If you know a good article, I would accept a reference as an answer, just as I would accept a "self-made" answer!

• – Dave May 28 at 0:14

1 Answer

Why should I choose one over the other?

You sould prepare a common validation set out of your dataset and try out each and every method on your dataset.

Following are the methods that I know to handle imbalanced datasets. -

1. Use weighted cross-entropy loss (as you mentioned)

• You can assign weights to your loss such that it will penalize more to the smaller classes and the less to larger classes. Many frameworks have a very easy way to do this.
• In Scikit-learn you can look out for class_weight parameter. For eg - random forest
• Here is how you can use this in Pytorch
• Here is how you can use this in Keras
2. Use focal loss

• Originally proposed for object detection, but we can also use this for any other use case. Read more about it here
• Here is how you can use this in Pytorch for multi-class classification
• Here is how you can use this in Keras
3. Over Sampling and Under Sampling

• There are so many techniques in this, check out imblearn a dedicated library just to deal with imbalanced datasets.
4. Create a separate model for small classes

• If you have some classes that have very small number of instances, you can consider creating a separate classifier for these small classes (called small_classifier for eg). You can group together these small clases under a single class (called small_class for eg) so that your main classifier will classify small_class with all other big classes in the dataset. And if your main classifier encounters any instance of small_class, it will pass it to small_classifier, which will predict the actual class for the small_class instance. This technique can give you accuracy boosts are now main classifier does not need to deal with very small classes, and insted small_classifier will be looking just at these small classes.
• Thank you! But one of the two approaches must in theory lead to better results than the other, right? – That Guy May 27 at 19:20
• I really appreciate your input but you do not seem to answer my main question but rather provide multiple methods for dealing with imbalanced classes in general – That Guy May 27 at 19:33
• Intuitively I agree with you, "I would think that using as many training examples as possible is the better option". But, I think it depends a lot on your dataset and problem statement, so I always try to test all of the methods on a common validation set. – Devashish Prasad May 28 at 4:49