0
$\begingroup$

Transformers encoder layer

Basically here the $Q$,$K$ and $V$ are passed through a linear layer to obtain the actual $Q$,$K$ and $V$ for self attention mechanism and then we concatenate all of it.

My doubt is, I thought the $Q$,$K$ and $V$ were obtained through the input embedding $X$.

$$Q=XW_q$$ $$K=XW_k$$ $$V=XW_v$$

How come we are using the $Q$,$K$ and $V$ and linearly projecting them to again get back $Q$,$K$ and $V$.

Sorry if my doubt is stupid!

$\endgroup$
0

1 Answer 1

2
$\begingroup$

We should not mistake the K, Q and V vectors received by the multi-head attention block with those received by the scaled dot-product block.

The K, Q and V vectors that are fed to the multi-head attention block are projected separately to a lower dimensional space for each of the attention heads so that each scaled-dot product can compute a different result. The dimension of the lower space is the original one divided by the number of heads.

After the scaled dot-product, the results of the individual scaled dot-products are combined back into a single vector, recovering the original dimensionality.

Only in the first attention layer, the values of the vectors fed to the multi-head attention block come from the embeddings. From the second layer on, the inputs come from the outputs of the previous layer.

$\endgroup$
6
  • $\begingroup$ So, are you saying that in a Mukti-head attention, Each attention head will receive Q K and V thats linearly projected down so that when computing the scaled dot product, The resulting sum will be different ? If so, All this while what I assumed was, for the first attention layer, We multiply the input X with Wq,Wv,Wk to obtain the first set of Q,K and V and for different heads within the same layer, We multiply the X with different Wq,Wv,Wk. I am assuming this is wrong? Like the same Q,K and V are projected down for different attention heads right? If so, Whats exactly wrong in my approach? $\endgroup$ May 14, 2023 at 17:03
  • $\begingroup$ And you said int he subsequent layers, We use the previous layers output instead of input embeddings. The subsequent layers also require a Wq,Wv and Wk matrix which should be initialised right? $\endgroup$ May 14, 2023 at 17:08
  • $\begingroup$ Yes, each attention layer has their own W matrices $\endgroup$
    – noe
    May 14, 2023 at 17:26
  • $\begingroup$ When you say "Whats exactly wrong in my approach?", what approach are you referring to? $\endgroup$
    – noe
    May 14, 2023 at 17:26
  • $\begingroup$ If so, All this while what I assumed was, for the first attention layer, We multiply the input X with Wq,Wv,Wk to obtain the first set of Q,K and V and for different heads within the same layer, We multiply the X with different Wq,Wv,Wk. $\endgroup$ May 14, 2023 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.