oversampling data with subclass

Question

Oversampling of under-represented data is a way to combat class imbalance. For example, if we have a training data set with 100 data points of class A and 1000 data points of class B, we can over sample the 100 A data (may be with some sophisticated oversampling methods) to generate 1000 A data to mitigate the data imbalance.

Now, let's say we have 1100 data points of class B, and class A has 2 subclasses, A1 and A2, which have 100 and 10 data points, respectively. And we are still interested in binary classification.

In this case, how should I over sample data of class A to address class imbalance? Should I over sample A1 to 1000 and A2 to 100, or over sample both A1 and A2 to 550?

Besides running an experiment, is there any theoretical analysis of this kind of class imbalance problem?

Answer 1

It totally depends on you and your problem. In fact, if the distribution of the data must be unchanged, oversampling is not a suitable solution. If you can change the Loss function of the algorithm, It will be very helpful. There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, and DP.

Disclaimer:

If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})  

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict      #the prediction of the implemented model

>>> y_actu = data.target        #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)

Answer 2

The theoretical analysis for this kind of problem would always be how to handle multiclass imbalance because even though it's a binary classification, you still want to oversample three classes.

Coming to your question of if you should over sample A1 and A2, the answer always depends on how well you know the data and if you know the ground truth of the data. I think it's really important to preserve the distribution of the classes when you roll them up together into one class.