The vanilla transformer decoder models proposed in the original paper "Attention Is All You Need" and the OpenAI GPT-series models are autoregressive models at inference time. In addition, I think the term "autoregressive" matters mostly for the inference behavior. In fact, primarily thanks to the masked attention mechanism, with appropriate caching optimization, one can only feed one token to and produce one token from a transformer decoder during inference. For more information, please check the analysis from my blog post "Transformer Autoregressive Inference Optimization".
The vanilla transformer decoder models are not trained in an autoregressive fashion, although it can be trained in such a way, just like the other classical language modeling models (such as recurrent neural networks) or time-series models. As Noe pointed out that we use golden tokens during training, instead of the tokens being predicted from the model that is being trained.
One can still train an autoregressive model using the predicted tokens if the user really wants to. But it would not make too much sense, especially when the model is not well trained at the beginning of the training. For example, if the user is trying to ask the model to learn language modeling of the sentence ["How", "are", "you", "?"]. But because the model is not mature, the autoregressive predictions starting from the first token become ["day", "wow", "oh", "haha", "bro", "."]. How would the user use these predicted tokens as inputs and correctly models the sentence ["How", "are", "you", "?"]? Specifically, in the vanilla transformer decoder, what the model would be learning would be:
- Given ["How"], predict ["day"].
- Given ["How", "are"], predict ["wow"].
- Given ["How", "are", "you"], predict ["oh"].
which absolutely makes no sense.
There are non-autoregressive models for sequence-to-sequence tasks. However, the modeling often requires much more sophisticated orchestration. Please refer to my blog post "Non-Autoregressive Model and Non-Autoregressive Decoding for Sequence to Sequence Tasks
" for details if you want.