I have been trying to find an answer to these questions, but I only find conflicting information. Is the transformer as a whole autoregressive or not? And what about the decoder? I understand that the decoder during inference proceeds autoregressively, but I am not sure about during training time.
Here are posts saying that the Transformer is not autoregressive:
Here are some saying that it is:
What would be the target input for Transformer Decoder during test phase?
https://www.tensorflow.org/text/tutorials/transformer
https://huggingface.co/transformers/summary.html#seq-to-seq-models