Bert: fine-tuning the entire pre-trained model end-to-end vs using contextual token vector

Question

In the official github page of BERT, it mentions that:

In certain cases, rather than fine-tuning the entire pre-trained model end-to-end, it can be beneficial to obtained pre-trained contextual embeddings, which are fixed contextual representations of each input token generated from the hidden layers of the pre-trained model. This should also mitigate most of the out-of-memory issues.

I am wondering in which cases, using only token vectors, will be more beneficial (other than out of memory issue)?

Answer 1

There are studies that explore the different cases where it is better fine-tuning versus directly using the vector representations. Maybe the most relevant study is To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks published at the ACL Conference in 2019.

Their results are summarized in Table 1, which, for BERT tells us that in general you obtain better results by fine-tuning (fire emoji 🔥) than with directly using the representations (ice emoji ❄️):

Answer 2

It goes like this: In the case of fine-tuning, you may want to use huge batch size, which may lead to the Out of Memory (OOM) issues. In such case, you may find the embeddings of BERT useful than fine-tuning on BERT itself. Maybe you can think of using BERT embeddings to train CNN or RNN classifier, in such case you can try to obtain the embeddings of BERT using a small batch size (it can be as low as 1), and then use these embeddings to further train your CNN or RNN classifier.