In the official github page of BERT, it mentions that:

In certain cases, rather than fine-tuning the entire pre-trained model end-to-end, it can be beneficial to obtained pre-trained contextual embeddings, which are fixed contextual representations of each input token generated from the hidden layers of the pre-trained model. This should also mitigate most of the out-of-memory issues.

I am wondering in which cases, using only token vectors, will be more beneficial (other than out of memory issue)?


It goes like this: In the case of fine-tuning, you may want to use huge batch size, which may lead to the Out of Memory (OOM) issues. In such case, you may find the embeddings of BERT useful than fine-tuning on BERT itself. Maybe you can think of using BERT embeddings to train CNN or RNN classifier, in such case you can try to obtain the embeddings of BERT using a small batch size (it can be as low as 1), and then use these embeddings to further train your CNN or RNN classifier.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.