I will use BERT's embedding weights (as discussed here) for embedding in embedding layers of the transformer model. But my question is: don't embeddings of BERT already go through the whole encoding layer and got that matrix? Why shouldn't I just remove-freeze the encoding layer and use BERT embedding vectors as input for the decoding layer? And also I will use BERT embeddings in the input of the decoding layer. Why should I not freeze attention layers in decoder layer too? Because embeddings of output text already have attention information?

  • 1
    $\begingroup$ What is your final objective? Do you want to fine-tune a Bert model or train one from scratch? $\endgroup$ Jun 23, 2022 at 18:59
  • $\begingroup$ @NicolasMartin I want to use pre-trained word embeddings and a pre-trained BERT encoder in transformers. $\endgroup$
    – canP
    Jun 23, 2022 at 19:12


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.