I am trying to wrap my head around how the Transformer architecture works. I think I have a decent top-level understanding of the encoder part, sort of how the Key, Query, and Value tensors work in the MultiHead attention layers. What I am struggling with is the decoder part, specifically the inputs to the very first decoder layer.

I understand that there are two things. The output of the final encoder layer, but before that an embedded (positional encoding + embedding) version of... well something.

In the original paper in Figure 1, they mention that the first decoder layer input is the Outputs (shifted right). I am a little confused on what they mean by "shifted right", but if I had to guess I would say the following is happening

Input: <Start> How are you <EOS> Output: <Start> I am fine <EOS>

and so the input to the first decoder layer will be [<Start> I am fine].

What is the need for shifting the sequence? Why would we not just input the target sequence itself? I was thinking maybe it's because of the auto-regressive nature of the decoder part, but then the only difference between the sequences would be the <EOS> token if I am seeing this correctly.

As you can probably tell I am a little bit confused by how some of the parts work, so any help to get to a better understanding would be much appreciated.


1 Answer 1


Following your example:

  • The source sequence would be How are you <EOS>
  • The input to the encoder would be How are you <EOS>. Note that there is no <start> token here.
  • The target sequence would be I am fine <EOS> . The output of the decoder will be compared against this in the training.
  • The input to the decoder would be <start> I am fine <EOS>.

Notice that the input to the decoder is the target sequence shifted one position to the right by the token that signals the beginning of the sentence. The logic of this is that the output at each position should receive the previous tokens (and not the token at the same position, of course), which is achieved with this shift together with the self-attention mask.

  • $\begingroup$ Thanks for your answer! Am I correct in understanding then that the Transformer will always output a sequence (I think that's probably self-explanatory but for some reason, I wasn't sure about it). My follow-up question would be, how is it that the decoder input and decoder output sequences will be of different lengths in the example you gave above? $\endgroup$
    – djvaroli
    Commented Feb 5, 2021 at 18:14
  • $\begingroup$ Yes, you are right, the sequence should actually be <start> I am fine. Note that in some implementations, instead of using a specific token for <start>, the <eos> token is reused, and in those cases the sequence would be <eos> I am fine. This is the case of fairseq; you can check their implementation, which is quite clear. $\endgroup$
    – noe
    Commented Feb 5, 2021 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.