# How do I handle class imbalance for text data when using pretrained models like BERT?

I have a skewed dataset consisting of samples of the form:

Category 1 10000
Category 2  2000
Category 3   400
Category 4   300
Category 5   100


The dataset consists of text with data labeled into one of the five categories. I am trying to use the pretrained models like BERT for the classification task but the model fails to identify the categories 3-5 .I have tried to apply class weights in the loss criterion however it doesn't help much although it gives better performance as compared to simple fine tuning of the pretrained models. I have came to know about SMOTE and other methods in order to handle the class imbalance issues . But since most of the transformer models expect the inputs as text which are later tokenized by their respective tokenizers I am not able to do any kind of oversampling . If there is a workaround for this thing I would be interested to know about it.

• Did you try downsampling majority class? – Ashwin Geet D'Sa Jan 6 at 16:59