0
$\begingroup$

I use this tutorial https://www.tensorflow.org/text/tutorials/classify_text_with_bert and get different accuracy depend on epoch numbers and batch sizes. What's optimal parameters?

$\endgroup$
1
$\begingroup$

The general answer is the same as with everything in machine learning: it depends on the particular task.

With Transformers, people tend to recommend larger batch sizes, typically thousands of tokens per batch. A highly cited paper on training tips for Transformers MT recommends getting the best results with 12k tokens per batch.

For the number of epochs, the usual advice is: plot the learning curves, at some point, the validation loss starts to stagnate or grow, whereas the training loss will continue to decrease. This means that the model starts overfitting. It will also tell what number of epochs is meaningful for your task.

$\endgroup$
1
  • $\begingroup$ Thanks. In my case validation loss starts grow, but accuracy still improve $\endgroup$ Nov 17 '21 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.