Is the Transformer decoder an autoregressive model?

Question

I have been trying to find an answer to these questions, but I only find conflicting information. Is the transformer as a whole autoregressive or not? And what about the decoder? I understand that the decoder during inference proceeds autoregressively, but I am not sure about during training time.

Here are posts saying that the Transformer is not autoregressive:

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

Here are some saying that it is:

What would be the target input for Transformer Decoder during test phase?

https://www.tensorflow.org/text/tutorials/transformer

https://towardsdatascience.com/illustrated-guide-to-transformers-step-by-step-explanation-f74876522bc0

https://huggingface.co/transformers/summary.html#seq-to-seq-models

Answer 1

The normal Transformer decoder is autoregressive at inference time and non-autoregressive at training time.

The non-autoregressive training can be done because of two factors:

1. We don't use the decoder's predictions as the next timestep input. Instead, we always use the gold tokens. This is referred to as teacher forcing.
2. The hidden states of all time steps are computed simultaneously in the attention heads. This is different in recurrent units (LSTMs, GRUs), where we need to have the previous timestep's hidden state to compute the current timestep's one.

At inference time, we don't have gold tokens, so we use the prediction of the decoder as the next timestep input.

Note that, despite being autoregressive at inference time, efficient implementations normally cache the hidden states of the previous timesteps, so they are not re-computed at each step.

As a whole, the Transformer model is autoregressive, because its decoder is autoregressive.

There are non-autoregressive variants of the Transformer (e.g. this), but they are more research topics than out-of-the-box solutions.