What is the value of Key, Value in the self attention calculation of Transformer model?
--> Every transformer block has these 3 set of weights: Query, Key, Value (Q, K, V). These weights determine the values. So, if the Q,K,V for each token is value obtained by multiplying it's embeddings with weights(Wq, Wk, Wv).
Query vector is embedding vector for the word that is queried, is that right?
--> Quite close, it's not the word embeddings, but value obtained by multiplying the embeddings of token (token embeddings + positional embeddings), with Query matrix.
Is attention calculated in RNN is different from self attention in Transformer?
--> Not much different, but they have little difference. In RNN, attentions are set of scalar weights learnt to determine which parts of inputs are more relevant. (by using a softmax you have sum of these attention weights equal to 1). Hover, in transformers attention is softmax over the value vectors obtained by multiplying query vector of a given token with the value vector of all the tokens. So, attention here is more to see which are the more relavent tokens for a single token.