0
$\begingroup$

I had a question related to SMOTE. If you have a data set that is imbalanced, is it correct to use SMOTE when you are using BERT? I believe I read somewhere that you do not need to do this since BERT take this into account, but I'm unable to find the article where I read that. Either from your own research or experience, would you say that oversampling using SMOTE (or some other algorithm) is useful when classifying using a BERT model? Or would it be redundant/unnecessary?

$\endgroup$
0
1
$\begingroup$

I don't know about any specific recommendation related to BERT, but my general advice is this:

  • Do not to systematically use oversampling when the data is imbalanced, at least not before specifically identifying performance issues caused by the imbalance. I see many questions here on DataScienceSE about solving problems which are caused by using oversampling blindly (and often wrongly).
  • In general resampling doesn't work well with text data, because language diversity cannot be simulated this way. There is a high risk to obtain an overfit model.
$\endgroup$
2
  • $\begingroup$ Basically, the model without doing something about the imbalanced classes can't really differentiate between 1 and 0. It shows that everything is 0 and nothing is 1. $\endgroup$ – QMan5 Mar 26 at 0:37
  • $\begingroup$ That's the usual symptom in case of data imbalance, but in general the real problem is why the model can't differentiate between the classes. Resampling would force the model to focus on the positive class but it likely wouldn't help with distinguishing between the classes, So the result is usually False Positive errors instead of False Negative errors. If the data doesn't contain any good indicator of the class, it's like putting a band-aid on a broken arm: a solution to the wrong problem. $\endgroup$ – Erwan Mar 26 at 9:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.