I am trying to wrap my head around how the Transformer architecture works. I think I have a decent top-level understanding of the encoder part, sort of how the Key, Query, and Value tensors work in the MultiHead attention layers. What I am struggling with is the decoder part, specifically the inputs to the very first decoder layer.
I understand that there are two things. The output of the final encoder layer, but before that an embedded (positional encoding + embedding) version of... well something.
In the original paper in Figure 1, they mention that the first decoder layer input is the Outputs (shifted right). I am a little confused on what they mean by "shifted right", but if I had to guess I would say the following is happening
Input: <Start> How are you <EOS> Output: <Start> I am fine <EOS>
and so the input to the first decoder layer will be [<Start> I am fine].
What is the need for shifting the sequence? Why would we not just input the target sequence itself? I was thinking maybe it's because of the auto-regressive nature of the decoder part, but then the only difference between the sequences would be the <EOS> token if I am seeing this correctly.
As you can probably tell I am a little bit confused by how some of the parts work, so any help to get to a better understanding would be much appreciated.