Imbalance classes problem

Question

I have a dataset of 4 classes with the following number of instances:

• Class 0: 13175
• Class 1: 82
• Class 2: 75
• Class 3: 121

Have have applied several subsampling and oversampling methods from the Python imbalance-learn API but none of them had a good performance for all classes. I have applied:

• Undersampling: CondensedNearestNeighbour, EditedNearestNeighbours, NeighbourhoodCleaningRule, RandomUnderSampler.
• Oversampling: SMOTE, ADASYN
• class_weight:['balanced'] parameter option in my grid search
• costcla library, but it does not work with more than two classes.

And I was not successful. Could you suggest a solution for this problem?

Answer 1

I think you're in a pretty difficult state. I think the class imbalance techniques you're using are fine. I would advise you to try framing your problem in a hierarchical classification manner.

Level 1 Classifier

classifier_A between class 0 & class (1 + 2 + 3)

Note: Apply all sort of class imbalance methods to get good accuracy here

Level 2 Classifier

classifier_B between class 1, class 2, class 3

Final pseudo model

if predict_classifier_A(x_test) == "class 0": result = "class 0" else: result = predict_classifier_B(x_test)

---

I'm not sure will it work or not but its worth a try. Let us know if it worked better.

Answer 2

If all above fails, I'd try this:

1. Fit model for each class. So you'd have four models, each specialized for that particular class.
2. I'd give a shot to any ensemble modeling technique - fit multiple models and combine them in a smart way.
3. Complex methods such as neural networks (I heard that GAN's are popular these days with impressive results). But that can be overkill.

In case non of that works, it could be just that your data cannot be separated in any way. I'd suggest to make some basic plots and calculate some simple statistics to get better understanding of the data and its separability.

Answer 3

Although, sampling techniques like up sampling, down sampling exist to solve this imbalance class issue. Smote sampling is also available in several packages. Another useful technique is to assign weights to the different observations (according to the frequencies in data) so that minor classes are not modelled as noise(usually minor classes are given more weights to instruct the model the the minor classes are valuable and not outliers or noise). Although the overall accuracy would be great for a classifier but the misclassification error is very high for the minor class. CARET package in R helps you prepare a basic framework to handle this class-imbalance problem. with CreateDataPartition function you can generate stratified datasets. Kindly explore this package in R documentation.

I would recommend to divide the four classes into chunks of equal sizes and make one dataset that would have equal representation of classes in one dataset. then the different datasets would have re-representation of minor classes but the dataset would have stratified sampling. These dataset can then be divided into training set and test set to do cross-validation. we can get the generalisation error by averaging out from the (say 10 fold). cross-validation. For second step for Hyperparameter tunning, internal cross fold validation can be performed to improve the prediction performance of the model.