3
$\begingroup$

In the official github page of BERT, it mentions that:

In certain cases, rather than fine-tuning the entire pre-trained model end-to-end, it can be beneficial to obtained pre-trained contextual embeddings, which are fixed contextual representations of each input token generated from the hidden layers of the pre-trained model. This should also mitigate most of the out-of-memory issues.

I am wondering in which cases, using only token vectors, will be more beneficial (other than out of memory issue)?

$\endgroup$
1
  • 1
    $\begingroup$ Please, consider upvoting the answers if you found them useful, and marking one of them as correct if deemed so. Alternatively, please considering describing what the answers are lacking or why you think they are not correct, so that they can be improved. $\endgroup$
    – noe
    May 9, 2021 at 23:31

2 Answers 2

1
$\begingroup$

There are studies that explore the different cases where it is better fine-tuning versus directly using the vector representations. Maybe the most relevant study is To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks published at the ACL Conference in 2019.

Their results are summarized in Table 1, which, for BERT tells us that in general you obtain better results by fine-tuning (fire emoji 🔥) than with directly using the representations (ice emoji ❄️):

enter image description here

$\endgroup$
0
$\begingroup$

It goes like this: In the case of fine-tuning, you may want to use huge batch size, which may lead to the Out of Memory (OOM) issues. In such case, you may find the embeddings of BERT useful than fine-tuning on BERT itself. Maybe you can think of using BERT embeddings to train CNN or RNN classifier, in such case you can try to obtain the embeddings of BERT using a small batch size (it can be as low as 1), and then use these embeddings to further train your CNN or RNN classifier.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.