I'm taking a course on Attention-based NLP but I'm not understanding the calculation and application of Attention, based on the use of Q, K, and V vectors. My understanding is that the K and V vectors are derived from the encoder input and the Q vector is derived from the decoder input. This makes sense to me in the context of training, where the entire input sequence is presented to the encoder and the entire output sequence is presented to the decoder. What does not make sense, however, is how this applies in the context of inference. In that case, it would seem like there is no input to the decoder, so where does the Q vector come from?
0
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Your understanding is correct: in the encoder-decoder attention blocks, the Keys and Values are the output of the encoder, while the Query vectors come from the decoder layers.
At inference time we have as many Query positions as the step we are in. Remember that at inference time, the decoder behaves autoregressive, meaning that at each timestep T it receives the T - 1 previous tokens and predicts the T token. Such a prediction is then concatenated to the previous step input and used as input for the following step. This way, in the first step, we only have one Query vector (per layer), which is the one belonging to the first position (the beginning of sequence token, aka <s> or <bos>). In the second step, we have two Query vectors, and so on.
