6
$\begingroup$

I want to fine tune BERT Multilingual but I'm not aware about the GPU requirements to train BERT Multilingual. I have GTX 1050ti 4GB on my local machine. I want to know what size of GPU is needed and what type of GPU is needed to train BERT. I have access to server resources. Could anyone tell me what size of GPU should I request for on server.

$\endgroup$

1 Answer 1

4
$\begingroup$

Speaking about vanilla BERT.

It is currently not possible to fine-tune BERT-Large using a GPU with 12GB - 16GB of RAM, because the maximum batch size that can fit in memory is too small (even with batch size = 1).

The fine-tuning examples which use BERT-Base should be able to run on a GPU that has at least 12GB of RAM using the hyperparameters given on this page.

However, GPU training is single-GPU only.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.