My question regards this image:

enter image description here

It seems that after the multi head attention there is a linear layer as they mention also from here: enter image description here

the linearity is given by the weights W^{o}. my quesion is: for the decoder, doesn't this linear layer mess up with the masking of the attention? I mean, if each token from the attention layer should not depend from the next tokens then these weights seem to mess up the whole thing or they don't? Indeed these linear weights will learn the dependencies among all the tokens and during inference there could be a problem, maybe(?). Thanks for any clarification.


1 Answer 1


No, this is not a problem.

If we zoom into the scaled dot product attention blocks, which happen before the projection with $W^O$ we see this:

enter image description here

There, you can see how the masking of the current and future positions happens inside the scaled dot product attention, which happens before the multiplication by $W^O$. Therefore, the values learned for $W^O$ are trained only with the information from the previous tokens in the decoder.

  • $\begingroup$ man, you are the god of transformers $\endgroup$
    – erre4
    Commented Mar 17, 2021 at 15:41
  • $\begingroup$ heheh, thanks, I am glad that the answer was helpful. $\endgroup$
    – noe
    Commented Mar 17, 2021 at 15:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.