So, in the decoder layer of transfomer, suppose I have predicted 3 words till now, including the start token then the last decoder layer will produce 3 vectors of size d-model, and only the last vector will pass through embedding layer to form logits. Am I getting this right? Because its nowhere mentioned in the original paper and I'm having a hard time understanding it. What about the information that gets lost by discarding the two tokens before the last token. We could try to linearly project all the vectors into a single d-dimension vector but then the size of vectors would keep on increasing after we predict new word everytime and we'd need a new projection matrix everytime. This detail seems implicit and isnt mentioned anywhere. Can someone provide me what is actually done and the reason behind this or is this a random heuristic that seems to work (i.e. just take the final hidden state produced by the decoder)
0
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I understand that we are talking about inference time (i.e. decoding), not training.
At each decoding step, all the predicted tokens are passed as input to the decoder, not only the last one. There is no information lost. The hidden states of the tokens that had already been decoded in the previous decoding steps are recomputed; however, non-naive implementations usually cache those hidden steps to avoid recomputing them over and over.
