5
$\begingroup$

Say I am using a maximum likelihood approach and my output unit computes a softmax function. My training set is distributed as follows over 6 classes:

class_samples[0]=23, class_samples[1]=5, class_samples[2]=44, 
class_samples[3]=14, class_samples[4]=19, class_samples[5]=31

What should I do?

  1. use the training set as given above with a normalizing weight balancing(e.g. using sklearn.utils.class_weight.compute_class_weight).

  2. or should I simply use the minimum number of samples in a class(i.e. 5) to extract a balanced distribution of examples?

Why should I choose one over the other? Intuitively, I would think that using as many training examples as possible is the better option. However, I have tried to do some computations but I fail to show that usage of all examples with a normalizing weight balancing is better.

I have of course tried to do some heavy research but for some reason I cannot find the answer. If you know a good article, I would accept a reference as an answer, just as I would accept a "self-made" answer!

$\endgroup$
1
0
$\begingroup$

Why should I choose one over the other?

You sould prepare a common validation set out of your dataset and try out each and every method on your dataset.

Following are the methods that I know to handle imbalanced datasets. -

  1. Use weighted cross-entropy loss (as you mentioned)

    • You can assign weights to your loss such that it will penalize more to the smaller classes and the less to larger classes. Many frameworks have a very easy way to do this.
    • In Scikit-learn you can look out for class_weight parameter. For eg - random forest
    • Here is how you can use this in Pytorch
    • Here is how you can use this in Keras
  2. Use focal loss

    • Originally proposed for object detection, but we can also use this for any other use case. Read more about it here
    • Here is how you can use this in Pytorch for multi-class classification
    • Here is how you can use this in Keras
  3. Over Sampling and Under Sampling

    • There are so many techniques in this, check out imblearn a dedicated library just to deal with imbalanced datasets.
  4. Create a separate model for small classes

    • If you have some classes that have very small number of instances, you can consider creating a separate classifier for these small classes (called small_classifier for eg). You can group together these small clases under a single class (called small_class for eg) so that your main classifier will classify small_class with all other big classes in the dataset. And if your main classifier encounters any instance of small_class, it will pass it to small_classifier, which will predict the actual class for the small_class instance. This technique can give you accuracy boosts are now main classifier does not need to deal with very small classes, and insted small_classifier will be looking just at these small classes.
$\endgroup$
3
  • $\begingroup$ Thank you! But one of the two approaches must in theory lead to better results than the other, right? $\endgroup$
    – That Guy
    May 27 at 19:20
  • 1
    $\begingroup$ I really appreciate your input but you do not seem to answer my main question but rather provide multiple methods for dealing with imbalanced classes in general $\endgroup$
    – That Guy
    May 27 at 19:33
  • 1
    $\begingroup$ Intuitively I agree with you, "I would think that using as many training examples as possible is the better option". But, I think it depends a lot on your dataset and problem statement, so I always try to test all of the methods on a common validation set. $\endgroup$ May 28 at 4:49
0
$\begingroup$

There is not enough data samples for machine learning. Most likely, any model trained on so few samples will not be able to generalize.

You should collect more data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.