So, I have a doubt in Attention is all you need:
The implementation of transformers on tensorflow's official documentation says:
Each multi-head attention block gets three inputs; Q (query), K (key), V (value). These are put through linear (Dense) layers and split up into multiple heads.
However, The paper mentions:
Instead of performing a single attention function with dmodel-dimensional keys, values and queries, we found it beneficial to linearly project the queries, keys and values h times with different, learned linear projections to dk, dk and dv dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding dv-dimensional output values.
There is no mention of splitting Q,K and V to obtain heads. Instead, the paper says that they are passed through 'h' different dense layers to convert d-model dimensional vectors to 'h' different dk,dk and dv dimensional vectors respectively. So basically the pseudocode, from what I understood, should look something like this:
Q,K & V are d-model dimensional vectors. for i in range(h): Qi = Dense(dk)(Q) Ki = Dense(dk)(K) Vi = Dense(dv)(V) Ai = Attention(Qi, Ki, Vi) A0, A1, A2, ..., Ah are then concatenated.
Is this right? or am I missing something here?