8
$\begingroup$

Does ChatGPT use an encoder-decoder architecture, or a decoder-only architecture? I have been coming across Medium and TowardsDataScience articles suggesting that it has an encoder-decoder architecture (see sources below):

-- https://medium.com/@sreedevr/chatgpt-architecture-explained-7fc0cbb7426c

-- https://towardsdatascience.com/how-chatgpt-works-the-models-behind-the-bot-1ce5fca96286

-- https://leoneperdigao.medium.com/chatgpt-a-deep-dive-1feade9c4d77

I even asked ChatGPT myself whether it uses an encoder and decoder, or just a decoder, and it told me it has both an encoder and decoder. Here is one of its responses to the question:

"As a language model, ChatGPT does not have a decoder-only architecture, it has an encoder-decoder architecture. Having an encoder-decoder architecture enables language models like ChatGPT to process input sequences and generate outputs based on that input. The encoder processes the input sequence and generates a set of hidden states, which are then used by the decoder to generate the output sequence. Having both an encoder and a decoder allows ChatGPT to handle a variety of tasks that involve generating text based on input text, such as language translation, summarization, and answering questions. Having an encoder-decoder architecture allows ChatGPT to maintain context and consistency in the generated output, which is important for many NLP tasks. It is possible to train a decoder-only language model, but it would be limited in its ability to generate coherent text as it would not have access to the context provided by the encoder."

However, I have been under the definite impression for quite some time now that GPT-3 (from which ChatGPT was in part derived) is a decoder-only model. And I take with a grain of salt ChatGPT's explanation of its own architecture given that it seems prone to generating incorrect answers sometimes. Also, with the huge fanfare of ChatGPT and the potential for misinformed authors writing about the model, I wonder if anyone knows of a reliable source that can clarify this question.

Thanks

$\endgroup$
8
  • $\begingroup$ I'm still not convinced by the current responses, to be honest. But I'm willing to be convinced if there is enough evidence from sources SPECIFICALLY regarding ChatGPT. Yes, I know that GPT2 and GPT3 are decoder-only language models. Yet, nowhere have I found any articles specifically mentioning ChatGPT is a decoder-only model. Yes, it makes sense that if ChatGPT has been fine-tuned on InstructGPT/GPT 3.5, which was in turn fine-tuned on GPT-3, that ChatGPT may very well also be a decoder-only. But without some official or otherwise credible statement about ChatGPT, I can't be certain. $\endgroup$
    – user141493
    Feb 3 at 17:33
  • $\begingroup$ According to the OpenAI website, ChatGPT is a fine-tuned version of GPT-3.5 which means it has the same architecture. What makes you think it does not? $\endgroup$
    – noe
    Feb 3 at 17:37
  • $\begingroup$ This is a quote from the ChatGPT presentation page: "ChatGPT is fine-tuned from a model in the GPT-3.5 series". $\endgroup$
    – noe
    Feb 3 at 17:40
  • $\begingroup$ Yeah, you may very well be right. I'm not too familiar with the fine-tuning process as far as whether a fine-tuned version of a model definitely needs to take the same architecture of the original (although it does make sense that it would), or whether there are techniques of fine-tuning where the architecture has been somewhat modified as well. $\endgroup$
    – user141493
    Feb 3 at 17:48
  • 1
    $\begingroup$ Traditionally MT is done with encoder-decoder. There are examples of decoder-only NMT like this one, but that's not the norm. Language models typically are not use that way but, as seen with ChatGPT, they also can translate reliably given the right type and amount of training data. $\endgroup$
    – noe
    Feb 3 at 18:04

5 Answers 5

14
+50
$\begingroup$
Summary

ChatGPT is the fine-tuning of GPT-3.5, which is a language model based on a Transformer decoder with some modifications with respect to the original Transformer architecture. Therefore it is a decoder-only model.

Complete information with references

The origin of ChatGPT was GPT (Generative pre-Trained Transformer). The evolution from GPT to ChatGPT was as follows:

  • GPT (see the OpenAI announcement) was a normal Transformer decoder. From the GPT paper:

    In our experiments, we use a multi-layer Transformer decoder [34] for the language model [...]

  • GPT-2 (see the OpenAI announcement and the source code) is also a Transformer decoder, but with some modifications. It is also bigger and trained on more data. From the GPT-2 paper:

    We use a Transformer (Vaswani et al., 2017) based architecture for our LMs. The model largely follows the details of the OpenAI GPT model (Radford et al., 2018) with a few modifications. Layer normalization (Ba et al., 2016) was moved to the input of each sub-block, similar to a pre-activation residual network (He et al., 2016) and an additional layer normalization was added after the final self-attention block. A modified initialization which accounts for the accumulation on the residual path with model depth is used. We scale the weights of residual layers at initialization by a factor of $1/\sqrt{N}$ where $N$ is the number of residual layers. The vocabulary is expanded to 50,257. We also increase the context size from 512 to 1024 tokens and a larger batch size of 512 is used.

  • GPT-3 is GPT-2 scaled up and with some modifications. From the GPT-3 paper published at NeurIPS'20:

    We use the same model and architecture as GPT-2 [RWC+19], including the modified initialization, pre-normalization, and reversible tokenization described therein, with the exception that we use alternating dense and locally banded sparse attention patterns in the layers of the transformer, similar to the Sparse Transformer [CGRS19]. To study the dependence of ML performance on model size, we train 8 different sizes of model, from 125 million parameters to 175 billion parameters, with the last being the model we call GPT-3.

  • InstructGPT (see the paper) is a fine-tuned version of GPT-3. From the paper

    [...] we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning

  • GPT-3.5 (see OpenAI announcement) (text-davinci-003) is a fine-tuned version of InstructGPT. From the announcement:

    code-davinci-002 is a base model, so good for pure code-completion tasks

    text-davinci-002 is an InstructGPT model based on code-davinci-002

    text-davinci-003 is an improvement on text-davinci-002

  • ChatGPT (gpt-3.5-turbo*) is a GPT-3.5 fine-tuned on human instructions by Reinforcement Learning with Human Feedback (RLHF). From the OpenAI website:

    gpt-3.5-turbo-0301 is an improvement on text-davinci-003, optimized for chat

    From the ChatGPT presentation page:

    We trained this model using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT, but with slight differences in the data collection setup. We trained an initial model using supervised fine-tuning: human AI trainers provided conversations in which they played both sides—the user and an AI assistant. We gave the trainers access to model-written suggestions to help them compose their responses. We mixed this new dialogue dataset with the InstructGPT dataset, which we transformed into a dialogue format.

    To create a reward model for reinforcement learning, we needed to collect comparison data, which consisted of two or more model responses ranked by quality. To collect this data, we took conversations that AI trainers had with the chatbot. We randomly selected a model-written message, sampled several alternative completions, and had AI trainers rank them. Using these reward models, we can fine-tune the model using Proximal Policy Optimization. We performed several iterations of this process.

To follow in detail the GPT evolution, I recommend the article How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources.

$\endgroup$
6
  • 3
    $\begingroup$ Great answer! One thing (for other readers) to note for the GPT-2 source code, when you see github.com/openai/gpt-2/blob/master/src/encoder.py, it is the "byte-pair encoding" and not the neural net architecture encoder. $\endgroup$
    – alvas
    Apr 27 at 10:48
  • $\begingroup$ Thanks for the clarification @alvas, the name encoder.py is indeed confusing in the context of this question. $\endgroup$
    – noe
    Apr 27 at 10:51
  • $\begingroup$ @alvas thanks a lot for the bounty! $\endgroup$
    – noe
    May 8 at 10:20
  • $\begingroup$ But there is a question in my mind. If it is decoder-only, then how does it deal with the input without an encoder. $\endgroup$ Jun 3 at 16:40
  • $\begingroup$ @random_hooman the prompt is given to the decoder itself. If you have further questions or need more detail, please create a new question. $\endgroup$
    – noe
    Jun 3 at 16:43
1
$\begingroup$

GPT stands for Generative Pretrained Transformer.

ChatGPT is a fine tuning of GPT.

Therefore, it is a Decoder.

$\endgroup$
0
$\begingroup$

Amazing how much dissonance exists on this very basic question-- are GPT2 based LLM's (GPT3 and ChatGPT) encoder-decoder or decoder only models? Googled ranked answers are all over the map with most converging on the responses here that they are "decoder only" models.

Well...... you can't decode data without first starting with encoded data. So the answer is.... OpenAI's GPT2 and subsequent models are encoder-decoder, but are commonly referred to as decoder models because the decoder part (right side of diagram) makes them distinct from the seq2seq -> BERT model evolutionary path from which they descended. But that does not mean they are "decoder only" or lack an encoder. It means that in contrast to BERT models which are "encoder only," they also invoke decoding.

  • All the above hypothesis only.

architecture

$\endgroup$
1
  • 1
    $\begingroup$ Of course that you can decode without an encoder. That's precisely what language models do. The "prompt" or initial text is given to the decoder itself as the prefix to start decoding from. You can check the source code of GPT-2, which is open source, to check by yourself that it is a decoder-only model. $\endgroup$
    – noe
    Mar 1 at 19:00
0
$\begingroup$

This is not a complete answer, but I think ChatGPT's response to the same question as the OP was asking is interesting, (1) to highlight that it can change it's answer and even be inconsistent within the same answer, and (2) the text at the end of the following answer seems like a reasonable explanation for why it said what it did before:

ChatGPT is a type of language model that uses a transformer architecture, which includes both an encoder and a decoder. Specifically, GPT-3, the model on which ChatGPT is based, uses a transformer decoder architecture without an explicit encoder component. However, the transformer decoder can be thought of as both an encoder and a decoder, since it receives input and generates output, similar to the way that an encoder-decoder architecture works. Therefore, even though ChatGPT does not have an explicit encoder component, it can be considered to have both encoding and decoding capabilities within its transformer architecture.

I appreciate the debate going on in the other answers. I think we need to get into semantics.

There are three generally-accepted classes of transformer LLM architectures: encoder, decoder, and encoder-decoder. Within this overly-simplistic taxonomy, the OpenAI GPT family of LLMs seem to best fit in the decoder branch because they do not have a full encoder like BERT, so they can't do things like extract structured data, e.g. named entity recognition, directly.

On the other hand, no language model can understand language without some form of encoding. At a minimum, the LM needs to convert words into word embeddings. This could be considered a form of encoding, though a very minimal one. I suspect ChatGPT does much more than word embeddings, so it would have an even less-than-minimal encoder.

But I'm not an expert, so I can't give details here ... yet.

$\endgroup$
0
$\begingroup$

The T5 paper (section 3.2.1) has a very readable discussion of various Transformer model structures. It says

As such, a Transformer decoder (without an encoder) can be used as a language model (LM), i.e. a model trained solely for next-step prediction (Liu et al., 2018; Radford et al., 2018; Al-Rfou et al., 2019).

where Radford et al., 2018 is the GPT paper.

As the T5 paper explains, "decoder" refers to a stack of Transformer layers with masked attention that cannot look ahead to future tokens, like the decoder in the original Transformer paper. It does not imply the presence of a side input from an encoder, because there is no encoder.

According to noe's answer, this answer for GPT should carry over to the original question on ChatGPT.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.