Training is not stable with extreme class imbalance

Question

I'm dealing with a multi-class classification problem with around 30 categories.

This problem has a severe class imbalance:

• Around 300 examples for the least common class.
• Around 100k examples for the most common class.

I don't want the classification model to be dummy and predict the most common class for most of the examples, for this reason, I'm using class_weight='balanced' in my LogisticRegression from sklearn. However, in this case, the classes that the algorithm predicts are mostly the less frequent ones. I understand the model overfits them somehow, as it assigns every sample from these class a very high weight.

On the other hand, if I don't apply the class weights, the model predicts the most common categories.

Is there a way to solve this? Is there a way to ensure the model predicts approximately the same proportion of samples for each category?

Answer 1

There are probably many different strategies but it's a difficult problem when the imbalance is as severe as it is here.

Without any correction the model is likely to ignore the smallest classes, as you noticed. However forcing the class weight as if the data is balanced is certainly too strong a correction. A middle ground would be to resample the training set instances yourself before fitting the model: by trying different ways to undersample the large classes and/or oversample the small classes you should be able to find an optimal tradeoff between the two extremes (use a separate validation set to determine the optimal combination).

Is there a way to ensure the model predicts approximately the same proportion of samples for each category?

Maybe I misunderstood but this looks like a bad idea: if the true proportions are not equal then the model shouldn't predict equal proportions either. The ideal scenario is for the model to predict the correct label every time, which implies predicting the true proportion for every class.

It might also be useful to analyze the performance in simpler configurations, e.g. by picking a few "average size" classes and observing how well the classifier discriminates between them only. The harder it is for a classifier to predict correctly, the more it relies on basic class proportion since it doesn't know any better.