The normal Transformer decoder is autoregressive at inference time and non-autoregressive at training time.
The non-autoregressive training can be done because of two factors:
- We don't use the decoder's predictions as the next timestep input. Instead, we always use the gold tokens. This is referred to as teacher forcing.
- The hidden states of all time steps are computed simultaneously in the attention heads. This is different in recurrent units (LSTMs, GRUs), where we need to have the previous timestep's hidden state to compute the current timestep's one.
At inference time, we don't have gold tokens, so we use the prediction of the decoder as the next timestep input.
Note that, despite being autoregressive at inference time, efficient implementations normally cache the hidden states of the previous timesteps, so they are not re-computed at each step.
As a whole, the Transformer model is autoregressive, because its decoder is autoregressive.
There are non-autoregressive variants of the Transformer (e.g. this), but they are more research topics than out-of-the-box solutions.