I am working on a multi-class text classification problem with hierarchical classes structure: super class and sub class for every text example. What am i trying to do is: based on the text predict the super class and then the sub class. To achieve this i do the following:
- fit one model to the data to predict the super class;
- then in a loop fit a separate model for each of the super classes to predict the sub class.
Unfortunately, some super classes have extremely small number of examples - up to 1 example in a class! Surely i cannot build a model predicting sub class on one example. The only thing one can do is to drop such super class from the data set. One can try building a model on 2 examples (1 - for train, 1 - for test sets and predict always the same unique sub class for this super class). Restricting the number of examples to 3, 5, 10, ... the model for such low-populated super classes will start making at least some sense.
Dealing with this problem of small number of examples is a trade-off between model usefulness and model performance. On one hand, i do need these low populated super classes, because when i get next real data set - there for sure will be examples from such 'trashy' super classes and I do want to be able to predict a sub class for such examples. On the other hand, I simply cannot apply the classifier to the data set because I either cannot properly split it into train and test set (ex.: super class has 100 examples, out of which 1 example is of sub class A and 99 examples are of sub class B) or model performance is low due to its disability to learn good on small number of cases.
The question is - what are typical approaches to solve such problem?
- are there any rules of thumb on the minimum (threshold) value for number of examples to have in a super class to built a more or less reliable model predicting its sub classes?
- should i apply oversampling for low-populated super/sub classes? And will it help to at least deal with train/test split issue?
- extra: suppose i drop examples with low-populated super classes. Then i predict probability on a new test set --> the model will give some (maybe high) probability that the example belongs to well-populated super class X, but in reality it belongs to some low-populated super class that i dropped on training stage. Should i just live with it? Or there is a way to determine from probability distributions across super classes that the model doing wrong?