3
$\begingroup$

Please see this image:

There are linear layers to modify the Query, key and value matrices and one linear layer after the multi head attention as they mention also from here:

Are these linear layers simply dense or fully connected layers? Let's consider the weight matrix Wi Q. Does this represent a dense layer with "Q" nodes? As they are using matrices as input rather than 1D vectors, I am getting a little confused.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

In the original Transformer article, these linear layers are just matrix multiplications.

As described in the paragraph you referred to in your question $W^Q$ is a matrix of dimensions $d_{model} \times d_k$, that is, it is a fully connected layer with $d_k$ units.

In practical implementations, these have the optional addition of a bias vector. You can see their actual definition in the fairseq code.

Improve this answer
$\endgroup$
3
  • $\begingroup$ In that case, are the weights of the linear layer same for each row of Q? $\endgroup$
    – fac120
    Commented May 20, 2021 at 16:10
  • $\begingroup$ Yes, for each attention head, the same $W^Q_i$ is applied to all query vectors in $Q$. $\endgroup$
    – noe
    Commented May 20, 2021 at 16:16
  • $\begingroup$ Thank you for all the help :) $\endgroup$
    – fac120
    Commented May 20, 2021 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.