ChatGPT's Architecture - Decoder Only? Or Encoder-Decoder?

Question

Does ChatGPT use an encoder-decoder architecture, or a decoder-only architecture? I have been coming across Medium and TowardsDataScience articles suggesting that it has an encoder-decoder architecture (see sources below):

-- https://medium.com/@sreedevr/chatgpt-architecture-explained-7fc0cbb7426c

-- https://towardsdatascience.com/how-chatgpt-works-the-models-behind-the-bot-1ce5fca96286

-- https://leoneperdigao.medium.com/chatgpt-a-deep-dive-1feade9c4d77

I even asked ChatGPT myself whether it uses an encoder and decoder, or just a decoder, and it told me it has both an encoder and decoder. Here is one of its responses to the question:

"As a language model, ChatGPT does not have a decoder-only architecture, it has an encoder-decoder architecture. Having an encoder-decoder architecture enables language models like ChatGPT to process input sequences and generate outputs based on that input. The encoder processes the input sequence and generates a set of hidden states, which are then used by the decoder to generate the output sequence. Having both an encoder and a decoder allows ChatGPT to handle a variety of tasks that involve generating text based on input text, such as language translation, summarization, and answering questions. Having an encoder-decoder architecture allows ChatGPT to maintain context and consistency in the generated output, which is important for many NLP tasks. It is possible to train a decoder-only language model, but it would be limited in its ability to generate coherent text as it would not have access to the context provided by the encoder."

However, I have been under the definite impression for quite some time now that GPT-3 (from which ChatGPT was in part derived) is a decoder-only model. And I take with a grain of salt ChatGPT's explanation of its own architecture given that it seems prone to generating incorrect answers sometimes. Also, with the huge fanfare of ChatGPT and the potential for misinformed authors writing about the model, I wonder if anyone knows of a reliable source that can clarify this question.

Thanks

Answer 1

Summary

ChatGPT is the fine-tuning of GPT-3.5, which is a language model based on a Transformer decoder with some modifications with respect to the original Transformer architecture. Therefore it is a decoder-only model.

Complete information with references

The origin of ChatGPT was GPT (Generative pre-Trained Transformer). The evolution from GPT to ChatGPT was as follows:

• GPT (see the OpenAI announcement) was a normal Transformer decoder. From the GPT paper:

  In our experiments, we use a multi-layer Transformer decoder [34] for the language model [...]

• GPT-2 (see the OpenAI announcement and the source code) is also a Transformer decoder, but with some modifications. It is also bigger and trained on more data. From the GPT-2 paper:

  We use a Transformer (Vaswani et al., 2017) based architecture for our LMs. The model largely follows the details of the OpenAI GPT model (Radford et al., 2018) with a few modifications. Layer normalization (Ba et al., 2016) was moved to the input of each sub-block, similar to a pre-activation residual network (He et al., 2016) and an additional layer normalization was added after the final self-attention block. A modified initialization which accounts for the accumulation on the residual path with model depth is used. We scale the weights of residual layers at initialization by a factor of $1/\sqrt{N}$ where $N$ is the number of residual layers. The vocabulary is expanded to 50,257. We also increase the context size from 512 to 1024 tokens and a larger batch size of 512 is used.

• GPT-3 is GPT-2 scaled up and with some modifications. From the GPT-3 paper published at NeurIPS'20:

  We use the same model and architecture as GPT-2 [RWC+19], including the modified initialization, pre-normalization, and reversible tokenization described therein, with the exception that we use alternating dense and locally banded sparse attention patterns in the layers of the transformer, similar to the Sparse Transformer [CGRS19]. To study the dependence of ML performance on model size, we train 8 different sizes of model, from 125 million parameters to 175 billion parameters, with the last being the model we call GPT-3.

• InstructGPT (see the paper) is a fine-tuned version of GPT-3. From the paper

  [...] we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning

• GPT-3.5 (see OpenAI announcement) (text-davinci-003) is a fine-tuned version of InstructGPT. From the announcement:

  code-davinci-002 is a base model, so good for pure code-completion tasks

  text-davinci-002 is an InstructGPT model based on code-davinci-002

  text-davinci-003 is an improvement on text-davinci-002

• ChatGPT (gpt-3.5-turbo*) is a GPT-3.5 fine-tuned on human instructions by Reinforcement Learning with Human Feedback (RLHF). From the OpenAI website:

  gpt-3.5-turbo-0301 is an improvement on text-davinci-003, optimized for chat

  From the ChatGPT presentation page:

  We trained this model using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT, but with slight differences in the data collection setup. We trained an initial model using supervised fine-tuning: human AI trainers provided conversations in which they played both sides—the user and an AI assistant. We gave the trainers access to model-written suggestions to help them compose their responses. We mixed this new dialogue dataset with the InstructGPT dataset, which we transformed into a dialogue format.

  To create a reward model for reinforcement learning, we needed to collect comparison data, which consisted of two or more model responses ranked by quality. To collect this data, we took conversations that AI trainers had with the chatbot. We randomly selected a model-written message, sampled several alternative completions, and had AI trainers rank them. Using these reward models, we can fine-tune the model using Proximal Policy Optimization. We performed several iterations of this process.

To follow in detail the GPT evolution, I recommend the article How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources.

Answer 2

GPT stands for Generative Pretrained Transformer.

ChatGPT is a fine tuning of GPT.

Therefore, it is a Decoder.

Answer 3

Amazing how much dissonance exists on this very basic question-- are GPT2 based LLM's (GPT3 and ChatGPT) encoder-decoder or decoder only models? Googled ranked answers are all over the map with most converging on the responses here that they are "decoder only" models.

Well...... you can't decode data without first starting with encoded data. So the answer is.... OpenAI's GPT2 and subsequent models are encoder-decoder, but are commonly referred to as decoder models because the decoder part (right side of diagram) makes them distinct from the seq2seq -> BERT model evolutionary path from which they descended. But that does not mean they are "decoder only" or lack an encoder. It means that in contrast to BERT models which are "encoder only," they also invoke decoding.

• All the above hypothesis only.

Answer 4

This is not a complete answer, but I think ChatGPT's response to the same question as the OP was asking is interesting, (1) to highlight that it can change it's answer and even be inconsistent within the same answer, and (2) the text at the end of the following answer seems like a reasonable explanation for why it said what it did before:

ChatGPT is a type of language model that uses a transformer architecture, which includes both an encoder and a decoder. Specifically, GPT-3, the model on which ChatGPT is based, uses a transformer decoder architecture without an explicit encoder component. However, the transformer decoder can be thought of as both an encoder and a decoder, since it receives input and generates output, similar to the way that an encoder-decoder architecture works. Therefore, even though ChatGPT does not have an explicit encoder component, it can be considered to have both encoding and decoding capabilities within its transformer architecture.

I appreciate the debate going on in the other answers. I think we need to get into semantics.

There are three generally-accepted classes of transformer LLM architectures: encoder, decoder, and encoder-decoder. Within this overly-simplistic taxonomy, the OpenAI GPT family of LLMs seem to best fit in the decoder branch because they do not have a full encoder like BERT, so they can't do things like extract structured data, e.g. named entity recognition, directly.

On the other hand, no language model can understand language without some form of encoding. At a minimum, the LM needs to convert words into word embeddings. This could be considered a form of encoding, though a very minimal one. I suspect ChatGPT does much more than word embeddings, so it would have an even less-than-minimal encoder.

But I'm not an expert, so I can't give details here ... yet.

Answer 5

The T5 paper (section 3.2.1) has a very readable discussion of various Transformer model structures. It says

As such, a Transformer decoder (without an encoder) can be used as a language model (LM), i.e. a model trained solely for next-step prediction (Liu et al., 2018; Radford et al., 2018; Al-Rfou et al., 2019).

where Radford et al., 2018 is the GPT paper.

As the T5 paper explains, "decoder" refers to a stack of Transformer layers with masked attention that cannot look ahead to future tokens, like the decoder in the original Transformer paper. It does not imply the presence of a side input from an encoder, because there is no encoder.

According to noe's answer, this answer for GPT should carry over to the original question on ChatGPT.

Answer 6

TLDR (simplified):

1. encoder sees into future, decoder predicts
2. transformer sees into future and then predicts, encodes, then decodes
3. gpt doesnt see into future, it only predicts - thats why it's decoder-only

you can check karpathy's explanation: https://youtu.be/kCc8FmEb1nY?t=6159 (~2m short, at the linked time, or for full understanding watch the whole 2h video, or even whole ~10h series)

basically, gpt decoder doesnt look into future, it is only predicting, so this is the decoding part.. and it is specific in that we dont know the full context of sentence..

however, in case of translating from one language to another, we know the full context from the original sentence, so basically before running the decoder to predict the next part, we first encode the context from the original language

to simplify, you could say, in this usecase, that encoding is more about encoding what the data means into symbolic representation, while decoding is more about predicting the output - the original transformer did both of these things, while gpt only does the second part

(to be clear, there is always some "encoding" in any code, but we are talking about transformer-style encoding when deciding whether to call gpt an encoder, and such encoding is not being done. but there are word/token embeddings being done in decoder, which you could call encoding, but that is just simple data processing. but in case of transformer, there is huge neural network that does encoding)