I want to fine tune BERT Multilingual but I'm not aware about the GPU requirements to train BERT Multilingual. I have GTX 1050ti 4GB on my local machine. I want to know what size of GPU is needed and what type of GPU is needed to train BERT. I have access to server resources. Could anyone tell me what size of GPU should I request for on server.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Speaking about vanilla BERT.
It is currently not possible to fine-tune BERT-Large using a GPU with 12GB - 16GB of RAM, because the maximum batch size that can fit in memory is too small (even with batch size = 1).
The fine-tuning examples which use BERT-Base should be able to run on a GPU that has at least 12GB of RAM using the hyperparameters given on this page.
However, GPU training is single-GPU only.
