# Transformers doubt

Basically here the $$Q$$,$$K$$ and $$V$$ are passed through a linear layer to obtain the actual $$Q$$,$$K$$ and $$V$$ for self attention mechanism and then we concatenate all of it.

My doubt is, I thought the $$Q$$,$$K$$ and $$V$$ were obtained through the input embedding $$X$$.

$$Q=XW_q$$ $$K=XW_k$$ $$V=XW_v$$

How come we are using the $$Q$$,$$K$$ and $$V$$ and linearly projecting them to again get back $$Q$$,$$K$$ and $$V$$.

Sorry if my doubt is stupid!

We should not mistake the K, Q and V vectors received by the multi-head attention block with those received by the scaled dot-product block.

The K, Q and V vectors that are fed to the multi-head attention block are projected separately to a lower dimensional space for each of the attention heads so that each scaled-dot product can compute a different result. The dimension of the lower space is the original one divided by the number of heads.

After the scaled dot-product, the results of the individual scaled dot-products are combined back into a single vector, recovering the original dimensionality.

Only in the first attention layer, the values of the vectors fed to the multi-head attention block come from the embeddings. From the second layer on, the inputs come from the outputs of the previous layer.

• So, are you saying that in a Mukti-head attention, Each attention head will receive Q K and V thats linearly projected down so that when computing the scaled dot product, The resulting sum will be different ? If so, All this while what I assumed was, for the first attention layer, We multiply the input X with Wq,Wv,Wk to obtain the first set of Q,K and V and for different heads within the same layer, We multiply the X with different Wq,Wv,Wk. I am assuming this is wrong? Like the same Q,K and V are projected down for different attention heads right? If so, Whats exactly wrong in my approach? May 14, 2023 at 17:03
• And you said int he subsequent layers, We use the previous layers output instead of input embeddings. The subsequent layers also require a Wq,Wv and Wk matrix which should be initialised right? May 14, 2023 at 17:08
• Yes, each attention layer has their own W matrices
– noe
May 14, 2023 at 17:26
• When you say "Whats exactly wrong in my approach?", what approach are you referring to?
– noe
May 14, 2023 at 17:26
• If so, All this while what I assumed was, for the first attention layer, We multiply the input X with Wq,Wv,Wk to obtain the first set of Q,K and V and for different heads within the same layer, We multiply the X with different Wq,Wv,Wk. May 14, 2023 at 17:28