# How to improve the result of f1 on imbalanced dataset

I have a dataset in which these are the distribution of the data:

Neutral.  15000
Negative  3000
positive  2000


And I am mostly interested to improve the performance on the negative category. I would say neutral and positive are not important for me. And I am using Bert model.

What I have tried so far:

1. undersample data: result was poor on negative category
2. Augment data with different approaches available in NLPaug. The result not only did not improve but it dropped by 4 percent
3. Class weight. Gave more weight to the negative class however did not affect the result and in some scenarios dropped
4. I tried to change the batch_size epoch etc... and it just had 0.5percent improvement

Now my question is that what could be the problem here? (is there anything I need to check in my dataset?)

And what else I can try to improve my model?, this is the general result I have so far

Negative 65
positive 72
neutral  90


And this is my confusion matrix:

               Pred_negative Pred_neutral Pred_positive
True_negative   138            101           3
True_neutral    53             1408          24
True_positive   2              25            69


I need to improve the negative category by at least 5 percent.

• If all you want to do is catch the negatives without regard for the other categories, why not call everything negative? Boom! You’ll have perfect ability to catch the negative cases, $100\%$ sensitivity/recall!
– Dave
Oct 9 at 5:28
• @Dave could you elaborate what do you mean please? if you mean why not have two class of negative positive (merging neutral and positive together) I have done that as well. and to my surprise the result did not improve!! Oct 9 at 5:47
• I mean that you predict every case as a negative, no matter what the features are. I suspect this extreme approach will not work for you, so what do you mean when you say that neutral and positive are not important? After all, you have a way to catch every negative case of you’re willing not to catch any neutral or positive cases.
– Dave
Oct 9 at 5:51
• Why would I want to predict a positive text as negative?? The negative is more important because of the business behind it. We want to be able to catch the negative reviews and don't care about neutral/positive. Oct 9 at 5:55
• You want to catch every negative text and don’t care about catching positive and neutral, right, so who cares of you misclassify neutral and positive texts? In other words, what do you lose by misclassifying the three types of texts? // It’s perfectly valid to decide that the cost of missing a negative text is so great that you should call everything negative to keep from missing any.
– Dave
Oct 9 at 6:05

• Thanks a lot for sharing your thoughts with me. Yes the evaluation is F1 score. I don't have any base line but I was hopeful that employing different technique like the ones mentioned could help me to reach to 70%. And regarding your suggestion, the weird thing is that even if when I make two categories positive negative still the performance does not change or drops Oct 9 at 22:16
• thanks so much for your input, it really helps me to go towards correct direction. By saying that the performance is the same I did not mean that its exactly the same but recall and precision changed however F1 score changes little. Is this possible that the problem is in data (especially in labeling? because we annotated data internally and as you know in some case it could be very subjective and the labeling could be different). because when I look at the confusion matrix I see many negative classes been predicted as neutral and some neutral class predicted as negative Oct 10 at 21:22