I am working on a multi-class text classification problem with hierarchical classes structure: super class and sub class for every text example. What am i trying to do is: based on the text predict the super class and then the sub class. To achieve this i do the following:

  • fit one model to the data to predict the super class;
  • then in a loop fit a separate model for each of the super classes to predict the sub class.

Unfortunately, some super classes have extremely small number of examples - up to 1 example in a class! Surely i cannot build a model predicting sub class on one example. The only thing one can do is to drop such super class from the data set. One can try building a model on 2 examples (1 - for train, 1 - for test sets and predict always the same unique sub class for this super class). Restricting the number of examples to 3, 5, 10, ... the model for such low-populated super classes will start making at least some sense.

Dealing with this problem of small number of examples is a trade-off between model usefulness and model performance. On one hand, i do need these low populated super classes, because when i get next real data set - there for sure will be examples from such 'trashy' super classes and I do want to be able to predict a sub class for such examples. On the other hand, I simply cannot apply the classifier to the data set because I either cannot properly split it into train and test set (ex.: super class has 100 examples, out of which 1 example is of sub class A and 99 examples are of sub class B) or model performance is low due to its disability to learn good on small number of cases.

The question is - what are typical approaches to solve such problem?

  • are there any rules of thumb on the minimum (threshold) value for number of examples to have in a super class to built a more or less reliable model predicting its sub classes?
  • should i apply oversampling for low-populated super/sub classes? And will it help to at least deal with train/test split issue?
  • extra: suppose i drop examples with low-populated super classes. Then i predict probability on a new test set --> the model will give some (maybe high) probability that the example belongs to well-populated super class X, but in reality it belongs to some low-populated super class that i dropped on training stage. Should i just live with it? Or there is a way to determine from probability distributions across super classes that the model doing wrong?
  • 1
    $\begingroup$ With a very low number of examples, it would be difficult to fit any model and evaluate how good it is. Oversampling may not help much, since there is not enough variation to learn. You can try using some domain knowledge to come up with rules for the really low population classes. Over time, as you accumulate more data, you can consider training models for these classes. $\endgroup$ – raghu Oct 30 '18 at 9:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.