In my current setup, I am trying to train a simple Bert model (DistilBert) for a classification task with 30 classes.

As is common, I performing a quick sanity check whether my code actually properly trains (tried an implementation with lightning)

After some 200 epochs, I get a loss of 0.029 where my loss function is Cross entropy.

My question arises from a quote that I picked out during the following lecture of the course Full Stack Deep Learning from Berkeley:


At some point it is mentioned that it is good practice to see if you can arbitrarily closely get the loss function to 0.

However, I implemented early stopping (patience 10) so I don't suspect that I will be able to push much farther past this value, meaning that I'm unsure whether my sanity check satisfied the arbitrarily close part.

Some interpretation, a cross entropy value of .029 corresponds to 93% of the weight to be in the correct class. Given that there are 30 classes, this might be satisfactory.

Any insights will be appreciated.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.