Please see this image:

There are linear layers to modify the Query, key and value matrices and one linear layer after the multi head attention as they mention also from here:

Are these linear layers simply dense or fully connected layers? Let's consider the weight matrix Wi Q. Does this represent a dense layer with "Q" nodes? As they are using matrices as input rather than 1D vectors, I am getting a little confused.


In the original Transformer article, these linear layers are just matrix multiplications.

As described in the paragraph you referred to in your question $W^Q$ is a matrix of dimensions $d_{model} \times d_k$, that is, it is a fully connected layer with $d_k$ units.

In practical implementations, these have the optional addition of a bias vector. You can see their actual definition in the fairseq code.

  • $\begingroup$ In that case, are the weights of the linear layer same for each row of Q? $\endgroup$
    – fac120
    May 20 '21 at 16:10
  • $\begingroup$ Yes, for each attention head, the same $W^Q_i$ is applied to all query vectors in $Q$. $\endgroup$
    – noe
    May 20 '21 at 16:16
  • $\begingroup$ Thank you for all the help :) $\endgroup$
    – fac120
    May 20 '21 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.