Rough ideas of expected performance boost from over-sampling techniques?

Question

I am trying to train a classifier for a multi class classification task. However, the dataset is very imbalanced. About half of the around 160 unique labels are such that there are only 10 or less samples corresponding to each of these rare labels. There are about 20 labels that occur exactly once. So the dataset contains a few classes that are well represented and very long and skinny tail of rare labels.

There are around 50 features (both numerical and categorical) and around 20 000 samples in the dataset.

I have looked briefly into this python library: https://imbalanced-learn.org/stable/index.html but I haven't yet gotten good results. Might be that I am not using it correctly.

What kind of boost from SMOTE or other methods for handling imbalanced dataset can I realistically expect? I know this is context dependent but trying to get some rough ideas.

Answer 1

My experience is oversampling with replacement may gives better classification performance than SMOTE on imbalanced data although the latter is considered more advanced than the former. If the minority classes are too small, the synthetic data generated by SMOTE can have wrong labels i.e. a synthetic instance is class 0 (a minority class) but should be class 1 (a majority class). This is because SMOTE labels the synthetic instances based on their K nearest neighbors in the training set. When the class 0 instances and the class 1 instances of the training set are mixed together in the multi-dimensional space of the features i.e. both classes can not be separated either linearly or non-linearly, SMOTE would label the synthetic instances wrong which leads to the poor classification performance of the trained classifier. Boosting can leads to over-fitting of ensemble model to the training set and poorer generalization than bagging. Because the base classifiers of boosting are trained one by one on those training instances which are classified wrong by the preceding base classifier. This over-fits to the training set.

The balanced random forest algorithm of imbalanced-learn library normally give promising results on imbalanced datasets. Balanced bagging classifier are worth trying.

https://imbalanced-learn.org/stable/references/generated/imblearn.ensemble.BalancedRandomForestClassifier.html

https://imbalanced-learn.org/stable/references/generated/imblearn.ensemble.BalancedBaggingClassifier.html#imblearn.ensemble.BalancedBaggingClassifier

Answer 2

The boost cannot be guessed in advanced..But what has been seen is that model start learning and gives much better performance so that model is acceptable in most cases if you have enough data and good training data.