Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Basically here the $Q$,$K$ and $V$ are passed through a linear layer to obtain the actual $Q$,$K$ and $V$ for self attention mechanism and then we concatenate all of it.
My doubt is, I thought the $Q$,$K$ and $V$ were obtained through the input embedding $X$.
$$Q=XW_q$$ $$K=XW_k$$ $$V=XW_v$$
How come we are using the $Q$,$K$ and $V$ and linearly projecting them to again get back $Q$,$K$ and $V$.
Sorry if my doubt is stupid!
We should not mistake the K, Q and V vectors received by the multi-head attention block with those received by the scaled dot-product block.
The K, Q and V vectors that are fed to the multi-head attention block are projected separately to a lower dimensional space for each of the attention heads so that each scaled-dot product can compute a different result. The dimension of the lower space is the original one divided by the number of heads.
After the scaled dot-product, the results of the individual scaled dot-products are combined back into a single vector, recovering the original dimensionality.
Only in the first attention layer, the values of the vectors fed to the multi-head attention block come from the embeddings. From the second layer on, the inputs come from the outputs of the previous layer.
