2
$\begingroup$

I am using pre-trained language model for binary classification. I fine-tune the model by training on data my downstream task. The results are good almost 98% F-measure.

However, when I remove a specific similar sentence from the training data and add it to my test data, the classifier fails to predict the class of that sentence. For example, the sentiment analysis task

"I love the movie more specifically the acting was great"

I removed from training all sentences containing the words " more specifically" and surprisingly in the test set they were all misclassified, so the precision decreased by a huge amount.

Any ideas on how can I further fine-tune/improve my model to work better on unseen text in training to avoid the problem I described above? (of course without feeding the model on sentences containing the words "more specifically")

Note: I observed the same performance regardless of the language model in use (BERT, RoBERTa etc).

$\endgroup$
1
  • $\begingroup$ Please consider marking one of the answers as correct ()with the tick mark ✓ if deemed so. Otherwise, please let us know what is not clear or why you think they are not correct. $\endgroup$
    – noe
    Dec 26, 2021 at 10:18

2 Answers 2

4
$\begingroup$

In order to make your model more robust to different wordings, you may try with data augmentation techniques, that is, creating variations of your sentences and adding them to the training set with the same label as the original sentence.

There are frameworks like TextAttack that offer several text augmentation techniques. Another option is using back-translation (i.e. translating your sentence into a second language and then translating that again into English), either locally with publicly available machine translation models or via some API like google translate.

Note that making fine-tuned language models resistant to this kind of (common) problems is an active area of research. For the latest advances, you can check this NeurIPS'21 article.

$\endgroup$
3
  • $\begingroup$ Thank you so much for your interesting answer and suggestion TextAttack. Weirdly as I mentioned above the problem seems that it is because of data augmentation (as it used to achieve 98~99% before removing specific sentence from the dataset). So I tried TextAttack sadly it only contributed to a fluctuating 1% increase. I also tried backtranslation from 3 language (Arabic, itlaian and French) but also no noticeable increase.. $\endgroup$
    – IS92
    Dec 26, 2021 at 11:09
  • $\begingroup$ I also tried multi-task fine-tuning (training on a very similar task but not the same) but same didn't achieve anything notable! However, more specifically were only misclassified 250 times, opposed to 700 when using the original train set, so maybe some sort of an ensemble would work but I am still not sure on how to design it, as multitask fine-tuning solves the problem of more specifically but overall the performance is lower than my original model. $\endgroup$
    – IS92
    Dec 26, 2021 at 11:16
  • $\begingroup$ I also tried manipulating the model itself freezing layers, adding my own layers after BERT but again to no avail. I will check the paper Thanks! $\endgroup$
    – IS92
    Dec 26, 2021 at 11:16
3
$\begingroup$

It appears that your model is failing to generalize.

One option is to increase the amount and quality of the training data.

Other options include large-scale language model specific regularization such as mixout and AUBER.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.