0
$\begingroup$

I have trained my BERT model(bert-base-cased) to detect toxic comments. I used the Toxic Comment Classification Challenge dataset from the Kaggle. My accuracy is 98% and the AUROC for various sub-classes is above 90%. However, my Precision, Recall, and F1 score is less. The scores are shown in the image Evaluation Scores. The dataset is highly imbalanced. The ratio of clean comments is way higher than the toxic comments. Any suggestions to improve the evaluation scores?

Here's the final score

           precision  recall  f1-score   support

micro avg 0.61 0.85 0.71 1743
macro avg 0.56 0.69 0.61 1743
weighted avg 0.64 0.85 0.72 1743
samples avg 0.08 0.09 0.08 1743

$\endgroup$
0
$\begingroup$

For imblanced data I recommend using false positive ratio instead of precision. In contrast to recall, precision is affected by the positive negative ratio, false-positive ratio (or 1 minus this number) is not affected by the data distribution, only by your model.

$\endgroup$
1
  • $\begingroup$ Thank you @Amit, I'll try using it for my evaluation. $\endgroup$ Aug 20 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.