How much data augmentation is required on an imbalanced dataset?

Question

Imagine I have a dataset with positive and negative sentences, and I need to train a transformer (Like BERT) to do the binary classification. The problem is that there are 100 negative sentences and 2000 positive sentences. There are libraries for NLP data augmentation like this one: https://github.com/makcedward/nlpaug

But how many new instances should I add to the minor class? Since my dataset is highly imbalanced, should I try to add 1900 instances to the minor class so that both classes have an equal population?

In order to make the imbalance ratio 1 in a highly imbalanced dataset such as mine, I have to use each sentence to generate 19 new sentences. If several of them are too alike, my model will end up overfitted.

Answer 1

There is nothing wrong with an imbalanced training dataset. It's possible that no changes are required.

When your training set is highly imbalanced like this, models in early training stages will predict everything to be the most prevalent class (positive in this case). After a longer training period, usually the model moves out of this local minima and starts making actual meaningful predictions. If it does, this is the ideal scenario because the model is using knowledge both of the text in the individual samples, but also of the class prevalences. Sometimes though, the model never escapes the local minima, and only then should you try to augment your training set.

Answer 2

Compute class weights for the labels in the dataset and then pass these weights to the loss function so that it takes care of the class imbalance. In PyTorch it can be done as shown below:

from sklearn.utils.class_weight import compute_class_weight

#compute the class weights
class_weights = compute_class_weight('balanced', np.unique(train_labels), train_labels)

print("Class Weights:",class_weights)

# converting list of class weights to a tensor
weights= torch.tensor(class_weights,dtype=torch.float)
    
# define the loss function
cross_entropy  = nn.NLLLoss(weight=weights)