Disclaimer : This is a long question, please be patient. Thanks in advance

I am using bert-base-uncased for text-classification. I have 11 classes, and the classification is happening alright for most of the classes. But of these 11 classes there are three classes, say A, B and C. Where there are high misclassification errors. I wish to reduce the errors between these classes.

Current State of my model :

  1. Model used Hugging Face bert-base-uncased.
  2. Loss function : Weighted Cross Entropy where the weights represent the inverse of the fraction of each class in the data.
  3. The text data related to classes A, B and C are not unbalanced and are roughly comparable to the most populous class

My Questions :

  1. Can anyone say why is this occuring?
  2. I am thinking of using some-other loss function specifically for these three classes, say soft-F1 from torchmetrics. The idea is that nn.Cross_Entropy() will be used for all classes and apart from that I will use soft-F1 when the true_label belongs to these three classes.

Thus the final loss function will be loss = frac * $nn.Cross_entropy() + (1-frac) * torchmetrics.soft_f1(if true_label in [A, B, C]), where frac is an hyper-parameter.

Will this approach work, or should I use something else ?

  • $\begingroup$ It could be because for the classes A, B C there are very few samples available to train the model on. $\endgroup$ Jul 31 at 11:22
  • $\begingroup$ No that is not the case, I mentioned that in the question. Each class has 3000+ examples and the whole data by itself is under 25000 examples. Thus they are well represented $\endgroup$ Jul 31 at 11:23
  • $\begingroup$ You can try making all the other classes as 0 and class A as 1 for one model, then do the same for other two classes. Basically making 3 models for each class and finally concatenate the results so you can get the final result. $\endgroup$ Jul 31 at 11:23
  • $\begingroup$ That is a very crude method, can you suggest some loss function based approach so that I can minimize that and increase F1 score $\endgroup$ Jul 31 at 11:31
  • $\begingroup$ try fbeta_score. You can tune it's values based on what you need. Example he parameter beta controls the trade-off between precision and recall: - If beta is 1, it is the F1 score, which equally weights precision and recall. - If beta is greater than 1, more emphasis is put on recall (capturing more true positives). - If beta is less than 1, more emphasis is put on precision (reducing false positives). $\endgroup$ Jul 31 at 11:42


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.