I want to to fine tune the BERT base model but the only accelerated HW I have access to is a couple of Quadro 600 GPUs which only pack 1GB RAM and 96 CUDA cores each.

My question: is it even possible to fine tune BERT over this HW? If so, would it be worth with so few CUDA cores available? How can I know whether GPU memory will fit all the data and parameters?

  • 2
    $\begingroup$ Nopes you can't fine-tune Bert with the above mentioned hardware (it depends on the seq length as well); better would be to extract the contextual embeddings and have something like Bi-Lstm maybe. $\endgroup$
    – Aditya
    Dec 30, 2019 at 3:41


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.