# How K and V are extracted from encoder output in transformer?

I was trying to understand transformer architecture from "Attention is all you need" paper. The paper shows following transformer architecture:

How $$K$$ and $$V$$ is extracted from $$512$$ dimensional encoder output (which is then fed to second multi head attention in decoder)?

• Please consider upvoting the answer if it was useful. Also, please consider accepting it if deemed correct, or alternatively, please indicate why you think it's not correct or not clear enough.
– noe
Commented Mar 28 at 19:20

Nothing needs to be extracted. The output of the encoder (a variable-length sequence of vectors of dimensionality $$d$$) is directly fed to the decoder's multi-head attention, playing the role of both key and value.
• But the output of the encoder is of dimension $d_{model}=512$ and key and value dimensions are $d_k=d_v=64$, right? By "playing the role of both", do you mean $K=V=\text{encoder output}$? Commented Apr 19, 2023 at 5:00
• Yes, K = V = encoder output, as shown in the transformer model diagram. Inside the multi-head attention block, each head applies a different projection $W_i^Q,W_i^K,W_i^V$ to map them to spaces with dimensionalities $d_k, d_k, d_v$ (see section 3.2.2 of the Attention is all you need article)