I have a skewed dataset consisting of samples of the form:

Category 1 10000
Category 2  2000
Category 3   400
Category 4   300 
Category 5   100

The dataset consists of text with data labeled into one of the five categories. I am trying to use the pretrained models like BERT for the classification task but the model fails to identify the categories 3-5 .I have tried to apply class weights in the loss criterion however it doesn't help much although it gives better performance as compared to simple fine tuning of the pretrained models. I have came to know about SMOTE and other methods in order to handle the class imbalance issues . But since most of the transformer models expect the inputs as text which are later tokenized by their respective tokenizers I am not able to do any kind of oversampling . If there is a workaround for this thing I would be interested to know about it.

  • $\begingroup$ Did you try downsampling majority class? $\endgroup$ – Ashwin Geet D'Sa Jan 6 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.