I print the weight names and shape of the BERT transformer. Now, I want to assign the printed weight to the layers in the transformers architecture: enter image description here

In the following, I can assign query, key and value:

enter image description here

But, in the print there are attention.output.dense.weight and attention.output.LayerNorm.weight, where can I find it in the architecture of transformer/attention-head?

Further there is an intermediate.dense.weight and there output.dense.weight and output.LayerNorm.weight. Are they parts of "Add & Norm" after the multi-head-attention blocks?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.