enter image description here

enter image description here

Those two pictures are from two similar experiments using same code.

I am fine-tuning a pretrained-Bert model to do a binary text classification task, the dataset is 50% positive vs 50% negative, so the classifier shouldn't classify everything as one class in the validation set as the picture shows.

I used AdamW optimizer with decreased learning rate.

I did the gradient clipping.

When I decrease the learning rate, it works fine, from 5e^-5 to 3e-5 or 2e-5.

What might be the problem here?

  • 2
    $\begingroup$ Something is incorrect in your code for sure. It's impossible to suggest anything without looking at it... $\endgroup$ – Aditya Aug 2 at 4:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.