0
$\begingroup$

I am working on a multi class text classifier. The total number of class that are there is 265 and total number of rows is 20,000. The class with largest number of occurrences has 6000 samples and there are many classes with exactly 1 sample as well. Preliminary analysis of the data led me to put a cutoff of 10 samples to be their for a class to be recognized and I made a separate miscellaneous class of less than 10 samples. Now I am reduced to 27 classes. And there is still class imbalance as can be seen from the figure with the class having largest number of samples has around 6000 and the lowest has 10.

How do I deal with such large class imbalance? Are their algorithms which are better suited to handle such large class imbalance?

Class frequency](htt[![enter image description here]1ps://imgur.com/a/8hJw9MJ)

$\endgroup$
1
$\begingroup$

For the most part, as you add more classes to your multiclass classification problem, it becomes more difficult to construct a model. All algorithms can run into trouble because there are fewer examples of each class to learn from. In case the 27 class target feature does not work well for you, you can decrease the number of classes even more. To do this:

  1. Continue what you already did by grouping small classes into the "other" category. Perhaps a model with 2 or 3 classes would be useful for you.
  2. Try a semi-supervised learning approach. Try clustering of your different categories and group similar categories into the same cluster. Your model will now predict cluster A, cluster B, etc.
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.