when a transformer model is trained there is linear layer in the end of decoder which i understand is a fully connected neural network. During training of a transformer model when a loss is obtained it will backpropagate to adjust the weights.

My question is how deep the backpropagation is?

  • does it happen only till linear layer weights(fully connected neural net) ?
  • OR does it extend to all the decoder layer weight matrices(Q,K,V) and Feed forward layers weights?
  • OR does it extend to the even the encoder + decoder weights ?

Please help me with the answer.


Backpropagation extends to the full model, through all decoder and encoder layers up to the embedding tables.

  • $\begingroup$ hence specifically i understand as weights backpropagated for 1.final linear layer weights, 2.decoder(Feed forward layer weights & Q, K ,V weight matrices), 3. encoder(Feed forward layer weights & Q, K ,V weight matrices) am i right? or am i missing out any other weights getting updated $\endgroup$
    – prog
    Feb 5 '21 at 15:14
  • $\begingroup$ You are missing the embeddings themselves. $\endgroup$
    – noe
    Feb 5 '21 at 15:49
  • $\begingroup$ thanks, out of curiosity i ask, does the same happens for bert too, i mean the update of embedding matrix during the pretraining process $\endgroup$
    – prog
    Feb 5 '21 at 16:18
  • 1
    $\begingroup$ Yes, it is the same for BERT. $\endgroup$
    – noe
    Feb 5 '21 at 16:25
  • $\begingroup$ thanks finally is there any reference /links on the mathematical update of these weights in transformer i can refer on ! $\endgroup$
    – prog
    Feb 5 '21 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.