I have a skewed dataset consisting of samples of the form:
Category 1 10000
Category 2 2000
Category 3 400
Category 4 300
Category 5 100
The dataset consists of text with data labeled into one of the five categories. I am trying to use the pretrained models like BERT for the classification task but the model fails to identify the categories 3-5 .I have tried to apply class weights in the loss criterion however it doesn't help much although it gives better performance as compared to simple fine tuning of the pretrained models. I have came to know about SMOTE and other methods in order to handle the class imbalance issues . But since most of the transformer models expect the inputs as text which are later tokenized by their respective tokenizers I am not able to do any kind of oversampling . If there is a workaround for this thing I would be interested to know about it.
