I have been trying to find an answer to these questions, but I only find conflicting information. Is the transformer as a whole autoregressive or not? And what about the decoder? I understand that the decoder during inference proceeds autoregressively, but I am not sure about during training time.

Here are posts saying that the Transformer is not autoregressive:

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

Here are some saying that it is:

What would be the target input for Transformer Decoder during test phase?




  • $\begingroup$ Please, consider marking the answer as correct if deemed so or, otherwise, specify what is not clear or missing in it. $\endgroup$
    – noe
    Nov 18, 2021 at 8:22

1 Answer 1


The normal Transformer decoder is autoregressive at inference time and non-autoregressive at training time.

The non-autoregressive training can be done because of two factors:

  1. We don't use the decoder's predictions as the next timestep input. Instead, we always use the gold tokens. This is referred to as teacher forcing.
  2. The hidden states of all time steps are computed simultaneously in the attention heads. This is different in recurrent units (LSTMs, GRUs), where we need to have the previous timestep's hidden state to compute the current timestep's one.

At inference time, we don't have gold tokens, so we use the prediction of the decoder as the next timestep input.

Note that, despite being autoregressive at inference time, efficient implementations normally cache the hidden states of the previous timesteps, so they are not re-computed at each step.

As a whole, the Transformer model is autoregressive, because its decoder is autoregressive.

There are non-autoregressive variants of the Transformer (e.g. this), but they are more research topics than out-of-the-box solutions.

  • $\begingroup$ "The normal Transformer decoder is autoregressive at inference time and non-autoregressive at training time." That's the essence. Very well put into the words! $\endgroup$ May 11 at 20:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.