While padding is normally applied after the normal tokens (i.e. right padding), it is perfectly fine to apply it before normal tokens (i.e. left padding). For instance, fairseq supports parameter left_pad to specify precisely this.

For left padding to be handled correctly, you must mask the padding tokens, because the self-attention mask would not prevent the hidden states of the normal token positions to depend on the padding positions.

About the need for tgt_key_padding_mask

While padding is usually applied after the normal tokens (i.e. right padding), it is perfectly fine to apply it before normal tokens (i.e. left padding). For instance, fairseq supports parameter left_pad to specify precisely this.

For left padding to be handled correctly, you must mask the padding tokens, because the self-attention mask would not prevent the hidden states of the normal token positions to depend on the padding positions.

About the meaning of each argument

I think the best documentation is in MultiHeadAttention.forward:

• key_padding_mask – if provided, specified padding elements in the key will be ignored by the attention. When given a binary mask and a value is True, the corresponding value on the attention layer will be ignored. When given a byte mask and a value is non-zero, the corresponding value on the attention layer will be ignored
• attn_mask – 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all the batches while a 3D mask allows to specify a different mask for the entries of each batch.

With that information and knowing where keys, values and queries come from in each multi-head attention block, it should be clear the purpose of each parameter in nn.TransformerDecoder.forward. Also, the documentation of MultiheadAttention.forward contains info about the expected shapes.