0
$\begingroup$

In my current setup, I am trying to train a simple Bert model (DistilBert) for a classification task with 30 classes.

As is common, I performing a quick sanity check whether my code actually properly trains (tried an implementation with lightning)

After some 200 epochs, I get a loss of 0.029 where my loss function is Cross entropy.

My question arises from a quote that I picked out during the following lecture of the course Full Stack Deep Learning from Berkeley:

https://www.youtube.com/watch?v=f1JRFu7X-c8

At some point it is mentioned that it is good practice to see if you can arbitrarily closely get the loss function to 0.

However, I implemented early stopping (patience 10) so I don't suspect that I will be able to push much farther past this value, meaning that I'm unsure whether my sanity check satisfied the arbitrarily close part.

Some interpretation, a cross entropy value of .029 corresponds to 93% of the weight to be in the correct class. Given that there are 30 classes, this might be satisfactory.

Any insights will be appreciated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.