I have a dataset in which these are the distribution of the data:

Neutral.  15000
Negative  3000
positive  2000

And I am mostly interested to improve the performance on the negative category. I would say neutral and positive are not important for me. And I am using Bert model.

What I have tried so far:

  1. undersample data: result was poor on negative category
  2. Augment data with different approaches available in NLPaug. The result not only did not improve but it dropped by 4 percent
  3. Class weight. Gave more weight to the negative class however did not affect the result and in some scenarios dropped
  4. I tried to change the batch_size epoch etc... and it just had 0.5percent improvement

Now my question is that what could be the problem here? (is there anything I need to check in my dataset?)

And what else I can try to improve my model?, this is the general result I have so far

Negative 65
positive 72
neutral  90

And this is my confusion matrix:

               Pred_negative Pred_neutral Pred_positive
True_negative   138            101           3
True_neutral    53             1408          24
True_positive   2              25            69

I need to improve the negative category by at least 5 percent.

  • $\begingroup$ If all you want to do is catch the negatives without regard for the other categories, why not call everything negative? Boom! You’ll have perfect ability to catch the negative cases, $100\%$ sensitivity/recall! $\endgroup$
    – Dave
    Oct 9 at 5:28
  • $\begingroup$ @Dave could you elaborate what do you mean please? if you mean why not have two class of negative positive (merging neutral and positive together) I have done that as well. and to my surprise the result did not improve!! $\endgroup$
    – Maria
    Oct 9 at 5:47
  • $\begingroup$ I mean that you predict every case as a negative, no matter what the features are. I suspect this extreme approach will not work for you, so what do you mean when you say that neutral and positive are not important? After all, you have a way to catch every negative case of you’re willing not to catch any neutral or positive cases. $\endgroup$
    – Dave
    Oct 9 at 5:51
  • 1
    $\begingroup$ Why would I want to predict a positive text as negative?? The negative is more important because of the business behind it. We want to be able to catch the negative reviews and don't care about neutral/positive. $\endgroup$
    – Maria
    Oct 9 at 5:55
  • $\begingroup$ You want to catch every negative text and don’t care about catching positive and neutral, right, so who cares of you misclassify neutral and positive texts? In other words, what do you lose by misclassifying the three types of texts? // It’s perfectly valid to decide that the cost of missing a negative text is so great that you should call everything negative to keep from missing any. $\endgroup$
    – Dave
    Oct 9 at 6:05

A few thoughts:

  • The evaluation method is not clear, in particular what are the evaluation scores shown, is it f1 score?
  • Why do you need to improve "by at least 5%"? Do you know the results of another system on the same data? If not it doesn't really make sense to aim for a particular performance value: performance depends a lot on the data, it's possible that your system already reaches the maximum performance with this dataset for example. You should at least have a baseline system to compare to, for example a basic Naive Bayes classifier.
  • One thing you could try is to remove the neutral category, this might help the model focus on the difference between negative and positive instead of trying to correctly classify the neutral category.
  • $\begingroup$ Thanks a lot for sharing your thoughts with me. Yes the evaluation is F1 score. I don't have any base line but I was hopeful that employing different technique like the ones mentioned could help me to reach to 70%. And regarding your suggestion, the weird thing is that even if when I make two categories positive negative still the performance does not change or drops $\endgroup$
    – Maria
    Oct 9 at 22:16
  • 2
    $\begingroup$ @Maria it's important to have a baseline system because the interpretation of the performance values depends on the data. It's possible that the performance would be the same with only two classes, it's not really meaningful. $\endgroup$
    – Erwan
    Oct 10 at 11:55
  • $\begingroup$ thanks so much for your input, it really helps me to go towards correct direction. By saying that the performance is the same I did not mean that its exactly the same but recall and precision changed however F1 score changes little. Is this possible that the problem is in data (especially in labeling? because we annotated data internally and as you know in some case it could be very subjective and the labeling could be different). because when I look at the confusion matrix I see many negative classes been predicted as neutral and some neutral class predicted as negative $\endgroup$
    – Maria
    Oct 10 at 21:22
  • 1
    $\begingroup$ @Maria annotating the sentiment of a text can be subjective so it's possible that several annotators would not annotate the same kind of text in the same way. The way to test this is to check inter-annotator agreement on the labelled data. It's not necessarily a problem because the data can be ambiguous, but it's a useful information because it's an upper bound for the performance of the model (if humans can't agree what is the class, chances are that the model can't find it reliably either). $\endgroup$
    – Erwan
    Oct 11 at 8:47
  • 1
    $\begingroup$ @Maria yes, this kind of error clearly affects the performance in general and the perf of the negative class in particular: here if these 28 instances were correctly labelled as negative, that would be 28 more correctly classified as negative and 28 less wrongly classified as neutral. So for the negative class you would have 166 TP instead of 138 and 27 FN instead of 55, so the recall would go from 0.71 to 0.86. The precision would also increase a bit so your f1-score for the negative class would be at least 0.70. $\endgroup$
    – Erwan
    Oct 12 at 11:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.