What are the advantages of autoregressive over seq2seq?

Question

Why are recent dialog agents, such as ChatGPT, BlenderBot3, and Sparrow, based on the decoder architecture instead of the encoder-decoder architecture?

I know the difference between the attention of the encoder and the decoder, but in terms of dialogue, isn't the attention of the encoder-decoder better?

Answer 1

Encoder-decoder architectures are normally used when there is an input sequence and an output sequence, and the output sequence is generated autoregressively. The encoder processes the whole input sequence at once, while the decoder receives the representations computed by the encoder and generates the output sequence autoregressively. The paradigmatical example is machine translation. To train an encoder-decoder model, you need pairs of input and output sequences.

Decoder-only architectures are normally used when you want to generate text autoregressively and there is no input (i.e. unconditional text generation), or when the input is the "prefix" of the output. The paradigmatical examples are language models. To train a decoder-only model, you need plain sequences.

While you can train a chatbot with an encoder-decoder architecture where the input is the user question or prompt and the output is the answer, this poses some problems:

• Difficulty to pre-train on massive text datasets scraped from the internet: large language models rely on being trained on large amounts of text downloaded from the internet. Encoder-decoder architectures need to have input and output sequences, which makes it more difficult to just feed whatever text is found on the internet as training data.
• Limited context: with encoder-decoder architectures, you need to define the input and output sequences to train the model. If you define them to be respectively the question/prompt from the user and the expected answer, then you are ignoring the previous questions and answers within the same conversation, which may contain key information to answer properly the following question. To properly use some hypothetical conversational training dataset making the model use the previous conversation as context, you would need, for every answer used as output, to give as input the whole previous conversation up to that moment. This is not practical. With decoder-only architectures you just feed the whole conversation to the model and that's it.

Apart from that, the computations of encoder-decoder attention are exactly the same as the decoder-only attention's, so no advantage there. In fact, it's been shown that using decoder-only architectures offers the same quality as encoder-decoder architectures, for machine translation at least.