My question is why the equally shaped attention head matrices $W^Q$, $W^K$, $W^V$ should not be the same $W = W^Q =W^K= W^V$. In my understanding of transformer-based language models one attention head is responsible for one syntactic or semantic relation between any two words in the context. One might think that such a relation is represented by one matrix $W$ that projects the full word embeddings $x_i$ from their full semantic space to a semantic subspace. Here we could - in principle - calculate scores $\sigma_{ij}$ as "similiarities" between two projected words $Wx_i$ and $Wx_j$ and then calculate the weighted sum of the projected tokens $Wx_k$.

I wonder why this would not work, and why we need three different matrices.

The other way around: What does it mean to calculate the score as the dot-product of two vectors from two different semantic subspaces? Is this still some kind of similiarity (which lies at the heart of word embeddings)? And doesn't it sound like comparing apples and pears?

  • $\begingroup$ Note that these "apples" and "pears" are not fixed, but learned. So are the $W$ matrices. Regardless of human semantic interpretations of what those operations mean or if they make sense, during training those weights are adjusted to be useful to improve the loss. $\endgroup$
    – noe
    Jun 15, 2023 at 8:14
  • $\begingroup$ @noe: But it was a design decision. What could have learned only one matrix. (Compare to the final linear layer which sometimes is the same as the word embedding layer.) $\endgroup$ Jun 15, 2023 at 9:01


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.