2

To start with your last question: you correctly say that BERT is an encoder-only model trained with the masked language-modeling objective and operates non-autoregressively. GPT-2 is a decode-only model trained using the left-to-right language objective and operates autoregressively. Other than that, there are only technical differences in hyper-parameters, ...


1

The normal Transformer decoder is autoregressive at inference time and non-autoregressive at training time. The non-autoregressive training can be done because of two factors: We don't use the decoder's predictions as the next timestep input. Instead, we always use the gold tokens. This is referred to as teacher forcing. The hidden states of all time steps ...


Only top voted, non community-wiki answers of a minimum length are eligible