Imbalanced training set vs smaller balanced training set?

Question

Say I am using a maximum likelihood approach and my output unit computes a softmax function. My training set is distributed as follows over 6 classes:

class_samples[0]=23, class_samples[1]=5, class_samples[2]=44, 
class_samples[3]=14, class_samples[4]=19, class_samples[5]=31

What should I do?

1. use the training set as given above with a normalizing weight balancing(e.g. using sklearn.utils.class_weight.compute_class_weight).

2. or should I simply use the minimum number of samples in a class(i.e. 5) to extract a balanced distribution of examples?

Why should I choose one over the other? Intuitively, I would think that using as many training examples as possible is the better option. However, I have tried to do some computations but I fail to show that usage of all examples with a normalizing weight balancing is better.

I have of course tried to do some heavy research but for some reason I cannot find the answer. If you know a good article, I would accept a reference as an answer, just as I would accept a "self-made" answer!

Answer 1

There are not enough data samples for machine learning. Most likely, any model trained on so few samples will not be able to generalize.

You should collect more data.

Answer 2

Why should I choose one over the other?

You sould prepare a common validation set out of your dataset and try out each and every method on your dataset.

Following are the methods that I know to handle imbalanced datasets. -

1. Use weighted cross-entropy loss (as you mentioned)

   • You can assign weights to your loss such that it will penalize more to the smaller classes and the less to larger classes. Many frameworks have a very easy way to do this.
   • In Scikit-learn you can look out for class_weight parameter. For eg - random forest
   • Here is how you can use this in Pytorch
   • Here is how you can use this in Keras

2. Use focal loss

   • Originally proposed for object detection, but we can also use this for any other use case. Read more about it here
   • Here is how you can use this in Pytorch for multi-class classification
   • Here is how you can use this in Keras

3. Over Sampling and Under Sampling

   • There are so many techniques in this, check out imblearn a dedicated library just to deal with imbalanced datasets.

4. Create a separate model for small classes

   • If you have some classes that have very small number of instances, you can consider creating a separate classifier for these small classes (called small_classifier for eg). You can group together these small clases under a single class (called small_class for eg) so that your main classifier will classify small_class with all other big classes in the dataset. And if your main classifier encounters any instance of small_class, it will pass it to small_classifier, which will predict the actual class for the small_class instance. This technique can give you accuracy boosts are now main classifier does not need to deal with very small classes, and insted small_classifier will be looking just at these small classes.