what is the first input to the decoder in a transformer model?

Question

The image is from url: Jay Alammar on transformers

K_encdec and V_encdec are calculated in a matrix multiplication with the encoder outputs and sent to the encoder-decoder attention layer of each decoder layer in the decoder.
The previous output is the input to the decoder from step 2 but what is the input to the decoder in step 1? Just the K_encdec and V_encdec or is it necessary to prompt the decoder by inputting the vectorized output (from the encoder) for the first word?

Answer 1

At each decoding time step, the decoder receives 2 inputs:

• the encoder output: this is computed once and is fed to all layers of the decoder at each decoding time step as key ($K_{endec}$) and value ($V_{endec}$) for the encoder-decoder attention blocks.
• the target tokens decoded up to the current decoding step: for the first step, the matrix contains in its first position a special token, normally </s>. After each decoding step $k$, the result of the decoder at position $k$ is written to the target tokens matrix at position $k+1$, and then the next decoding step takes place.

For instance, in the fairseq implementation of the decoding, you can see how they create the target tokens matrix and fill it with padding here and then how they place an EOS token (</s>) at the first position here.

As you have tagged your question with the bert tag, you should know that what I described before only applies to the sequence-to-sequence transduction task way of using the Transformer (i.e. when used for machine translation), and this is not how BERT works. BERT is trained on a masked language model loss which makes its use at inference time much different than the NMT Transformer.