I am trying to understand the transformer architecture.

I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example take 8 heads.

Now for a particular layer we will have 8 different sets of (Wq, Wk, Wv), the weight matrices used to calculate the query, key and value.

Now what I want to know is whether these weight matrices are shared between the different layers i.e are the (Wq, Wk, Wv) matrices of head#1 in layer 1 same for head#1 of layers 2, 3, ....?

And if they are shared, doesn't it affect in parallelization?

  • 1
    $\begingroup$ The heads are not shared among layers. $\endgroup$
    – Astariul
    Commented Dec 9, 2019 at 2:24
  • 1
    $\begingroup$ If with layer 1, you mean the first encoder stack, well Bert uses different sets of Wq, Wk, Wv, but Albert uses the same, so I think it is something that you choose. $\endgroup$ Commented Jun 28, 2021 at 3:06


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.