3
$\begingroup$

I have a dataset with around 70 classes and the dataset is largely balanced ~150 samples per class. I am finetuning RoBERTA-base for 4 epochs with a {lr =5e-5, wd = 0.01, batch_size=32}, so fairly standard hyperparameters.

Two of the classes in the dataset are "increaseCreditLimit" and "decreaseCreditLimit", and the data in them is quite obviously different. But when I test the model on very straightforward utterances like "increase my credit limit", "increase my credit card limit", etc, the prediction is always "decreaseCreditLimit".

A finetuned distilBERT gets all of them correct, so I know that there is nothing amiss in the dataset or the preprocessing. Can someone please shed some light on what could be wrong with the finetuning of the RoBERTa model?

If we look at the SHAP values, it seems that the RoBERTA tokenizer breaks the word "increase" into "incre" and "ase". But the distilBERT tokenizer doesn't do so and the attribution for the word "increase" is very significant.

RoBERTA SHAP values (Output 14 corresponds to the "increaseCreditLimit" class and Output 57 corresponds to "decreaseCreditLimit"

RoBERTA SHAP values (Output 14 corresponds to the "increaseCreditLimit" class and Output 57 corresponds to "decreaseCreditLimit")

DistilBERT SHAP values

DistilBERT SHAP values

Train val loss per epoch enter image description here

$\endgroup$
2
  • 1
    $\begingroup$ Can you show the fine-tuning training and validation loss curves? $\endgroup$
    – noe
    Commented Apr 24 at 6:41
  • $\begingroup$ updated the post. $\endgroup$ Commented Apr 24 at 10:26

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.