0
$\begingroup$

Disclaimer : This is a long question, please be patient. Thanks in advance

I am using bert-base-uncased for text-classification. I have 11 classes, and the classification is happening alright for most of the classes. But of these 11 classes there are three classes, say A, B and C. Where there are high misclassification errors. I wish to reduce the errors between these classes.

Current State of my model :

  1. Model used Hugging Face bert-base-uncased.
  2. Loss function : Weighted Cross Entropy where the weights represent the inverse of the fraction of each class in the data.
  3. The text data related to classes A, B and C are not unbalanced and are roughly comparable to the most populous class

My Questions :

  1. Can anyone say why is this occuring?
  2. I am thinking of using some-other loss function specifically for these three classes, say soft-F1 from torchmetrics. The idea is that nn.Cross_Entropy() will be used for all classes and apart from that I will use soft-F1 when the true_label belongs to these three classes.

Thus the final loss function will be loss = frac * $nn.Cross_entropy() + (1-frac) * torchmetrics.soft_f1(if true_label in [A, B, C]), where frac is an hyper-parameter.

Will this approach work, or should I use something else ?

Improve this question
$\endgroup$
10
  • $\begingroup$ It could be because for the classes A, B C there are very few samples available to train the model on. $\endgroup$
    – Harshad Patil
    Jul 31 at 11:22
  • $\begingroup$ No that is not the case, I mentioned that in the question. Each class has 3000+ examples and the whole data by itself is under 25000 examples. Thus they are well represented $\endgroup$
    – Vinay Varahabhotla
    Jul 31 at 11:23
  • $\begingroup$ You can try making all the other classes as 0 and class A as 1 for one model, then do the same for other two classes. Basically making 3 models for each class and finally concatenate the results so you can get the final result. $\endgroup$
    – Harshad Patil
    Jul 31 at 11:23
  • $\begingroup$ That is a very crude method, can you suggest some loss function based approach so that I can minimize that and increase F1 score $\endgroup$
    – Vinay Varahabhotla
    Jul 31 at 11:31
  • $\begingroup$ try fbeta_score. You can tune it's values based on what you need. Example he parameter beta controls the trade-off between precision and recall: - If beta is 1, it is the F1 score, which equally weights precision and recall. - If beta is greater than 1, more emphasis is put on recall (capturing more true positives). - If beta is less than 1, more emphasis is put on precision (reducing false positives). $\endgroup$
    – Harshad Patil
    Jul 31 at 11:42

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.