Highly Imbalanced dataset fro classes more than 200

Question

I have a text dataset where I need to train a classifier to classify the titles into categories. The dataset shape is more than 575000. There are 256 target classes here. The problem is the dataset is highly imbalanced. For target X1 it has 171793 records, X2 has 101575,........Xn-1 has 2, Xn has 2 records. Consider the target value counts are in decreasing order.

To handle with the imbalanced dataset, oversampling and undersampling works for multiclass say 3 classes. But in my case, there are 256 classes. How do I sample my dataset in this situation? How do I sample the dataset in a way so my model is stable for all the targets?

Do I have to remove the classes which have value counts 2 - 100 from this dataset? and apply undersampling/oversampling. Is there any approach to handle these type of situation?

Answer 1

The situation you describe is a few-shot learning problem: you have a lot of classes and only a few examples for some of them.

Similarity metric learning with siamese neural networks is well suited for this task. The idea is to learn a general similarity metric between examples, then classify new examples as belonging to the class of the "closest" sample from the training set. It seems a bit complex but it's probably the best way to learn "across" classes.

The link I've shared above is applied to an NLP problem so should be able to reuse it fairly easily.

Answer 2

Actually 200 isn't that much. It depends more how those are distributed. How do you extract features from raw text? For example: embedders give you fixed-size numeric vector and they're well suited for resampling.

No free lunch

You cannot state, that resampling might damage your decision space, unless you actually try it. I advise you do simple resampling with SMOTESVM followed by Tomek-Links. Run it and compare the metrics calculated in weighted mode. Then we'll decide what to do next.

Answer 3

Although sampling data or giving extra weight for handling imbalanced data-set are suggested, they're not good ways. I suggest you use an appropriate loss function in learning procedure for handling imbalanced data-set instead of sampling the abnormal class.

There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, and DP.

Disclaimer:

If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})  

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict      #the prediction of the implemented model

>>> y_actu = data.target        #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)