1
$\begingroup$

I am doing a NLP binary classification task, using Bert + softmax layer on top of it. The network uses cross-entropy loss.

When the ratio of positive class to negative class is 1:1 or 1:2, the model performs well on correctly classifying both classes(accuracy for each class is around 0.92).

When the ratio is 1:3 to 1:10, the model performs poorly as expected. When the ratio is 1:10, the model has a 0.98 accuracy on correctly classifying negative class instances, but only has a 0.80 accuracy on correctly classifying positive class instances.

The behavior is as expected as the model turns to classify most/all instances toward negative class since the ratio of positive class to negative class is 1:10.

I just want to ask what's the recommended way for handling this kind of class imbalance problem in natural language processing specifically?

I saw someone suggests to change loss function, or perform up/down sampling, but most of them are targetting computer vision class imbalance problem.

$\endgroup$
2
  • $\begingroup$ How is the performance obtained for the different ratios? Downsampling the negative instances? $\endgroup$
    – Erwan
    Mar 27 '21 at 12:29
  • $\begingroup$ @Erwan, yeah, I tried with the same number of positive instances, with a different number of negative instances. $\endgroup$
    – LGDGODV
    Mar 27 '21 at 21:17
0
$\begingroup$

Disclaimer: this answer might be disappointing ;)

In general my advice would be to carefully analyze the errors that the model makes and try to make the model deal with these cases better. This can involve many different strategies depending on the task and the data. Here are a few general directions to consider:

  • Most of the time the imbalance is not the real problem, the real problem is why the model can't differentiate between the classes. Even in case of extreme imbalance if the classes are easy to discriminate a model can perform very well. The imbalance only causes the model to assign the majority class when it doesn't have enough indication to decide, so it resorts to the conservative choice.
  • If the minority class is really small in absolute terms it's likely that there's not enough language diversity in the positive instances (data sparsity). This will usually cause a kind of overfitting which can be hidden by the fact that the model almost always assigns the majority class. In this case the goal should be to treat the overfitting, so the first direction is to simplify the model and/or the data representation.
  • Sometimes it can make sense to consider alternative ML designs: in a regular classification problem a model relies on the distribution of the classes, by principle. Some alternative approaches might not as influenced by the distribution, for example one-class classification. Of course it's not suited for every problem.

Overall my old-school advice is not to rely too much on technical answers such as resampling methods. It can make sense sometimes, but it shouldn't be used as some kind if magical answer instead of careful analysis.

$\endgroup$
2
  • $\begingroup$ You are right, I tried classifying texts that are semantically distant apart, even the ratio is 1:10, the model still performs very well. But my case is to classify two types of posts that are semantically close to each other, one-class classification performs poorly in this case. Do you think that change a loss function might improve much for my binary classifier? $\endgroup$
    – LGDGODV
    Mar 28 '21 at 5:33
  • $\begingroup$ @LGDGODV In your case I think it boils down to a precision/recall tradeoff: based on the performance values that you report, I assume that you have high precision but low recall with strong imbalance and conversely. So you can use the proportion as a parameter to tune the performance depending on the goal of the task, but I don't think there's a way to improve performance for both classes (unless you find a way to make the model better at classifiying the difficult cases, of course). $\endgroup$
    – Erwan
    Mar 28 '21 at 10:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.