0
$\begingroup$

I was trying to understand transformer architecture from "Attention is all you need" paper. The paper shows following transformer architecture:

enter image description here

How $K$ and $V$ is extracted from $512$ dimensional encoder output (which is then fed to second multi head attention in decoder)?

$\endgroup$
1
  • $\begingroup$ Please consider upvoting the answer if it was useful. Also, please consider accepting it if deemed correct, or alternatively, please indicate why you think it's not correct or not clear enough. $\endgroup$
    – noe
    Apr 27, 2023 at 20:05

1 Answer 1

0
$\begingroup$

Nothing needs to be extracted. The output of the encoder (a variable-length sequence of vectors of dimensionality $d$) is directly fed to the decoder's multi-head attention, playing the role of both key and value.

$\endgroup$
2
  • $\begingroup$ But the output of the encoder is of dimension $d_{model}=512$ and key and value dimensions are $d_k=d_v=64$, right? By "playing the role of both", do you mean $K=V=\text{encoder output}$? $\endgroup$
    – Mahesha999
    Apr 19, 2023 at 5:00
  • $\begingroup$ Yes, K = V = encoder output, as shown in the transformer model diagram. Inside the multi-head attention block, each head applies a different projection $W_i^Q,W_i^K,W_i^V$ to map them to spaces with dimensionalities $d_k, d_k, d_v$ (see section 3.2.2 of the Attention is all you need article) $\endgroup$
    – noe
    Apr 19, 2023 at 7:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.