I was trying to understand transformer architecture from "Attention is all you need" paper. It says following regarding dimensions of different vectors:
- The input consists of queries and keys of dimension $d_k$, and values of dimension $d_v$.
- $MultiHead(Q, K, V ) = Concat(head_1, ..., head_h) W^O$ where $head_i = Attention(QW_i^Q,KW^K_i,VW^V_i)$
where $W_i^Q\in\mathbb{R}^{d_{model}\times d_k}$, $W_i^K\in\mathbb{R}^{d_{model}\times d_k}$, $W_i^V\in\mathbb{R}^{d_{model}\times d_v}$, $W^O\in\mathbb{R}^{hd_{v}\times d_{model}}$- $h=8$ parallel attention layers or heads
- $d_k=d_v=d_{model}/h=64$
From these I figured out dimensions of vectors at different position in the transformers model as follows (in red colored text):
I have following doubts:
Is dimension of $K$ (vector of multiple/all keys that current word-query needs to attend to) $=d_k$? Or $d_k$ is just for single key? If for single key, then what is the dimension for $K$? Same is the doubt with $Q$ and $d_q$. I feel $Q$ is set of all queries that can apply to single word. $K$ is the set of all keys that single word can attend to. If that is the case, then dimensions of $Q$ must be $d_q\times\text{number of queries to consider}$ and $K$ must be $d_k\times\text{number of keys to attend for each word-query}$ But then what is this number of queries and keys?