0
$\begingroup$

I had a question related to SMOTE. If you have a data set that is imbalanced, is it correct to use SMOTE when you are using BERT? I believe I read somewhere that you do not need to do this since BERT take this into account, but I'm unable to find the article where I read that. Either from your own research or experience, would you say that oversampling using SMOTE (or some other algorithm) is useful when classifying using a BERT model? Or would it be redundant/unnecessary?

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
3
$\begingroup$

I don't know about any specific recommendation related to BERT, but my general advice is this:

  • Do not to systematically use oversampling when the data is imbalanced, at least not before specifically identifying performance issues caused by the imbalance. I see many questions here on DataScienceSE about solving problems which are caused by using oversampling blindly (and often wrongly).
  • In general resampling doesn't work well with text data, because language diversity cannot be simulated this way. There is a high risk to obtain an overfit model.
Improve this answer
$\endgroup$
2
  • $\begingroup$ Basically, the model without doing something about the imbalanced classes can't really differentiate between 1 and 0. It shows that everything is 0 and nothing is 1. $\endgroup$
    – QMan5
    Mar 26, 2021 at 0:37
  • $\begingroup$ That's the usual symptom in case of data imbalance, but in general the real problem is why the model can't differentiate between the classes. Resampling would force the model to focus on the positive class but it likely wouldn't help with distinguishing between the classes, So the result is usually False Positive errors instead of False Negative errors. If the data doesn't contain any good indicator of the class, it's like putting a band-aid on a broken arm: a solution to the wrong problem. $\endgroup$
    – Erwan
    Mar 26, 2021 at 9:50
0
$\begingroup$

Sorry for the delay in my response, but it is posible that others also benefit from it. I have had the same problem before. I would say an effective way to overcome the problem in most cases is to first try augmentation using some techniques from nlpaug and backtranslation from several languages see here and then drop duplicates.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.