What are the inputs to the first decoder layer in a Transformer model during the training phase?

Question

I am trying to wrap my head around how the Transformer architecture works. I think I have a decent top-level understanding of the encoder part, sort of how the Key, Query, and Value tensors work in the MultiHead attention layers. What I am struggling with is the decoder part, specifically the inputs to the very first decoder layer.

I understand that there are two things. The output of the final encoder layer, but before that an embedded (positional encoding + embedding) version of... well something.

In the original paper in Figure 1, they mention that the first decoder layer input is the Outputs (shifted right). I am a little confused on what they mean by "shifted right", but if I had to guess I would say the following is happening

Input: <Start> How are you <EOS> Output: <Start> I am fine <EOS>

and so the input to the first decoder layer will be [<Start> I am fine].

What is the need for shifting the sequence? Why would we not just input the target sequence itself? I was thinking maybe it's because of the auto-regressive nature of the decoder part, but then the only difference between the sequences would be the <EOS> token if I am seeing this correctly.

As you can probably tell I am a little bit confused by how some of the parts work, so any help to get to a better understanding would be much appreciated.

Answer 1

Following your example:

• The source sequence would be How are you <EOS>
• The input to the encoder would be How are you <EOS>. Note that there is no <start> token here.
• The target sequence would be I am fine <EOS> . The output of the decoder will be compared against this in the training.
• The input to the decoder would be <start> I am fine <EOS>.

Notice that the input to the decoder is the target sequence shifted one position to the right by the token that signals the beginning of the sentence. The logic of this is that the output at each position should receive the previous tokens (and not the token at the same position, of course), which is achieved with this shift together with the self-attention mask.