16
$\begingroup$

Does ChatGPT use an encoder-decoder architecture, or a decoder-only architecture? I have been coming across Medium and TowardsDataScience articles suggesting that it has an encoder-decoder architecture (see sources below):

-- https://medium.com/@sreedevr/chatgpt-architecture-explained-7fc0cbb7426c

-- https://towardsdatascience.com/how-chatgpt-works-the-models-behind-the-bot-1ce5fca96286

-- https://leoneperdigao.medium.com/chatgpt-a-deep-dive-1feade9c4d77

I even asked ChatGPT myself whether it uses an encoder and decoder, or just a decoder, and it told me it has both an encoder and decoder. Here is one of its responses to the question:

"As a language model, ChatGPT does not have a decoder-only architecture, it has an encoder-decoder architecture. Having an encoder-decoder architecture enables language models like ChatGPT to process input sequences and generate outputs based on that input. The encoder processes the input sequence and generates a set of hidden states, which are then used by the decoder to generate the output sequence. Having both an encoder and a decoder allows ChatGPT to handle a variety of tasks that involve generating text based on input text, such as language translation, summarization, and answering questions. Having an encoder-decoder architecture allows ChatGPT to maintain context and consistency in the generated output, which is important for many NLP tasks. It is possible to train a decoder-only language model, but it would be limited in its ability to generate coherent text as it would not have access to the context provided by the encoder."

However, I have been under the definite impression for quite some time now that GPT-3 (from which ChatGPT was in part derived) is a decoder-only model. And I take with a grain of salt ChatGPT's explanation of its own architecture given that it seems prone to generating incorrect answers sometimes. Also, with the huge fanfare of ChatGPT and the potential for misinformed authors writing about the model, I wonder if anyone knows of a reliable source that can clarify this question.

Thanks

$\endgroup$
8
  • $\begingroup$ I'm still not convinced by the current responses, to be honest. But I'm willing to be convinced if there is enough evidence from sources SPECIFICALLY regarding ChatGPT. Yes, I know that GPT2 and GPT3 are decoder-only language models. Yet, nowhere have I found any articles specifically mentioning ChatGPT is a decoder-only model. Yes, it makes sense that if ChatGPT has been fine-tuned on InstructGPT/GPT 3.5, which was in turn fine-tuned on GPT-3, that ChatGPT may very well also be a decoder-only. But without some official or otherwise credible statement about ChatGPT, I can't be certain. $\endgroup$
    – user141493
    Commented Feb 3, 2023 at 17:33
  • $\begingroup$ According to the OpenAI website, ChatGPT is a fine-tuned version of GPT-3.5 which means it has the same architecture. What makes you think it does not? $\endgroup$
    – noe
    Commented Feb 3, 2023 at 17:37
  • $\begingroup$ This is a quote from the ChatGPT presentation page: "ChatGPT is fine-tuned from a model in the GPT-3.5 series". $\endgroup$
    – noe
    Commented Feb 3, 2023 at 17:40
  • $\begingroup$ Yeah, you may very well be right. I'm not too familiar with the fine-tuning process as far as whether a fine-tuned version of a model definitely needs to take the same architecture of the original (although it does make sense that it would), or whether there are techniques of fine-tuning where the architecture has been somewhat modified as well. $\endgroup$
    – user141493
    Commented Feb 3, 2023 at 17:48
  • 1
    $\begingroup$ Traditionally MT is done with encoder-decoder. There are examples of decoder-only NMT like this one, but that's not the norm. Language models typically are not use that way but, as seen with ChatGPT, they also can translate reliably given the right type and amount of training data. $\endgroup$
    – noe
    Commented Feb 3, 2023 at 18:04

7 Answers 7

23
+50
$\begingroup$
Summary

ChatGPT is the fine-tuning of GPT-3.5, which is a language model based on a Transformer decoder with some modifications with respect to the original Transformer architecture. Therefore it is a decoder-only model.

Complete information with references

The origin of ChatGPT was GPT (Generative pre-Trained Transformer). The evolution from GPT to ChatGPT was as follows:

  • GPT (see the OpenAI announcement) was a normal Transformer decoder. From the GPT paper:

    In our experiments, we use a multi-layer Transformer decoder [34] for the language model [...]

  • GPT-2 (see the OpenAI announcement and the source code) is also a Transformer decoder, but with some modifications. It is also bigger and trained on more data. From the GPT-2 paper:

    We use a Transformer (Vaswani et al., 2017) based architecture for our LMs. The model largely follows the details of the OpenAI GPT model (Radford et al., 2018) with a few modifications. Layer normalization (Ba et al., 2016) was moved to the input of each sub-block, similar to a pre-activation residual network (He et al., 2016) and an additional layer normalization was added after the final self-attention block. A modified initialization which accounts for the accumulation on the residual path with model depth is used. We scale the weights of residual layers at initialization by a factor of $1/\sqrt{N}$ where $N$ is the number of residual layers. The vocabulary is expanded to 50,257. We also increase the context size from 512 to 1024 tokens and a larger batch size of 512 is used.

  • GPT-3 is GPT-2 scaled up and with some modifications. From the GPT-3 paper published at NeurIPS'20:

    We use the same model and architecture as GPT-2 [RWC+19], including the modified initialization, pre-normalization, and reversible tokenization described therein, with the exception that we use alternating dense and locally banded sparse attention patterns in the layers of the transformer, similar to the Sparse Transformer [CGRS19]. To study the dependence of ML performance on model size, we train 8 different sizes of model, from 125 million parameters to 175 billion parameters, with the last being the model we call GPT-3.

  • InstructGPT (see the paper) is a fine-tuned version of GPT-3. From the paper

    [...] we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning

  • GPT-3.5 (see OpenAI announcement) (text-davinci-003) is a fine-tuned version of InstructGPT. From the announcement:

    code-davinci-002 is a base model, so good for pure code-completion tasks

    text-davinci-002 is an InstructGPT model based on code-davinci-002

    text-davinci-003 is an improvement on text-davinci-002

  • ChatGPT (gpt-3.5-turbo*) is a GPT-3.5 fine-tuned on human instructions by Reinforcement Learning with Human Feedback (RLHF). From the OpenAI website:

    gpt-3.5-turbo-0301 is an improvement on text-davinci-003, optimized for chat

    From the ChatGPT presentation page:

    We trained this model using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT, but with slight differences in the data collection setup. We trained an initial model using supervised fine-tuning: human AI trainers provided conversations in which they played both sides—the user and an AI assistant. We gave the trainers access to model-written suggestions to help them compose their responses. We mixed this new dialogue dataset with the InstructGPT dataset, which we transformed into a dialogue format.

    To create a reward model for reinforcement learning, we needed to collect comparison data, which consisted of two or more model responses ranked by quality. To collect this data, we took conversations that AI trainers had with the chatbot. We randomly selected a model-written message, sampled several alternative completions, and had AI trainers rank them. Using these reward models, we can fine-tune the model using Proximal Policy Optimization. We performed several iterations of this process.

To follow in detail the GPT evolution, I recommend the article How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources.

$\endgroup$
7
  • 4
    $\begingroup$ Great answer! One thing (for other readers) to note for the GPT-2 source code, when you see github.com/openai/gpt-2/blob/master/src/encoder.py, it is the "byte-pair encoding" and not the neural net architecture encoder. $\endgroup$
    – alvas
    Commented Apr 27, 2023 at 10:48
  • $\begingroup$ Thanks for the clarification @alvas, the name encoder.py is indeed confusing in the context of this question. $\endgroup$
    – noe
    Commented Apr 27, 2023 at 10:51
  • $\begingroup$ @alvas thanks a lot for the bounty! $\endgroup$
    – noe
    Commented May 8, 2023 at 10:20
  • $\begingroup$ But there is a question in my mind. If it is decoder-only, then how does it deal with the input without an encoder. $\endgroup$ Commented Jun 3, 2023 at 16:40
  • $\begingroup$ @random_hooman the prompt is given to the decoder itself. If you have further questions or need more detail, please create a new question. $\endgroup$
    – noe
    Commented Jun 3, 2023 at 16:43
4
$\begingroup$

TLDR (simplified):

  1. encoder sees into future, decoder predicts
  2. transformer sees into future and then predicts, encodes, then decodes
  3. gpt doesnt see into future, it only predicts - thats why it's decoder-only

you can check karpathy's explanation: https://youtu.be/kCc8FmEb1nY?t=6159 (~2m short, at the linked time, or for full understanding watch the whole 2h video, or even whole ~10h series)

basically, gpt decoder doesnt look into future, it is only predicting, so this is the decoding part.. and it is specific in that we dont know the full context of sentence..

however, in case of translating from one language to another, we know the full context from the original sentence, so basically before running the decoder to predict the next part, we first encode the context from the original language

to simplify, you could say, in this usecase, that encoding is more about encoding what the data means into symbolic representation, while decoding is more about predicting the output - the original transformer did both of these things, while gpt only does the second part

(to be clear, there is always some "encoding" in any code, but we are talking about transformer-style encoding when deciding whether to call gpt an encoder, and such encoding is not being done. but there are word/token embeddings being done in decoder, which you could call encoding, but that is just simple data processing. but in case of transformer, there is huge neural network that does encoding)

$\endgroup$
1
$\begingroup$

GPT stands for Generative Pretrained Transformer.

ChatGPT is a fine tuning of GPT.

Therefore, it is a Decoder.

$\endgroup$
0
$\begingroup$

Amazing how much dissonance exists on this very basic question-- are GPT2 based LLM's (GPT3 and ChatGPT) encoder-decoder or decoder only models? Googled ranked answers are all over the map with most converging on the responses here that they are "decoder only" models.

Well...... you can't decode data without first starting with encoded data. So the answer is.... OpenAI's GPT2 and subsequent models are encoder-decoder, but are commonly referred to as decoder models because the decoder part (right side of diagram) makes them distinct from the seq2seq -> BERT model evolutionary path from which they descended. But that does not mean they are "decoder only" or lack an encoder. It means that in contrast to BERT models which are "encoder only," they also invoke decoding.

  • All the above hypothesis only.

architecture

$\endgroup$
1
  • 1
    $\begingroup$ Of course that you can decode without an encoder. That's precisely what language models do. The "prompt" or initial text is given to the decoder itself as the prefix to start decoding from. You can check the source code of GPT-2, which is open source, to check by yourself that it is a decoder-only model. $\endgroup$
    – noe
    Commented Mar 1, 2023 at 19:00
0
$\begingroup$

This is not a complete answer, but I think ChatGPT's response to the same question as the OP was asking is interesting, (1) to highlight that it can change it's answer and even be inconsistent within the same answer, and (2) the text at the end of the following answer seems like a reasonable explanation for why it said what it did before:

ChatGPT is a type of language model that uses a transformer architecture, which includes both an encoder and a decoder. Specifically, GPT-3, the model on which ChatGPT is based, uses a transformer decoder architecture without an explicit encoder component. However, the transformer decoder can be thought of as both an encoder and a decoder, since it receives input and generates output, similar to the way that an encoder-decoder architecture works. Therefore, even though ChatGPT does not have an explicit encoder component, it can be considered to have both encoding and decoding capabilities within its transformer architecture.

I appreciate the debate going on in the other answers. I think we need to get into semantics.

There are three generally-accepted classes of transformer LLM architectures: encoder, decoder, and encoder-decoder. Within this overly-simplistic taxonomy, the OpenAI GPT family of LLMs seem to best fit in the decoder branch because they do not have a full encoder like BERT, so they can't do things like extract structured data, e.g. named entity recognition, directly.

On the other hand, no language model can understand language without some form of encoding. At a minimum, the LM needs to convert words into word embeddings. This could be considered a form of encoding, though a very minimal one. I suspect ChatGPT does much more than word embeddings, so it would have an even less-than-minimal encoder.

But I'm not an expert, so I can't give details here ... yet.

$\endgroup$
0
$\begingroup$

The T5 paper (section 3.2.1) has a very readable discussion of various Transformer model structures. It says

As such, a Transformer decoder (without an encoder) can be used as a language model (LM), i.e. a model trained solely for next-step prediction (Liu et al., 2018; Radford et al., 2018; Al-Rfou et al., 2019).

where Radford et al., 2018 is the GPT paper.

As the T5 paper explains, "decoder" refers to a stack of Transformer layers with masked attention that cannot look ahead to future tokens, like the decoder in the original Transformer paper. It does not imply the presence of a side input from an encoder, because there is no encoder.

According to noe's answer, this answer for GPT should carry over to the original question on ChatGPT.

$\endgroup$
0
$\begingroup$

I asked my instance of ChatGPT 3.5 today (2023.12.18, state of training: January 2022) about encoder/decoder etc. and got this answer:

The GPT-3.5 architecture is a sophisticated model with 96 transformer layers, each containing 3072 nodes, resulting in a massive neural network. It follows an autoregressive transformer model, deviating from the traditional Encoder-Decoder structure and relying on a decoder-only setup.

With a vast number of transformer blocks, GPT-3.5 has an impressive scale, boasting 175 billion parameters. These parameters empower the model to comprehend and generate text across a diverse range of contexts, showcasing its versatility and depth of knowledge.

Trained on extensive datasets covering various topics, languages, and communication styles, the model can generate contextually relevant responses in numerous scenarios, from answering factual questions to engaging in creative and imaginative conversations.

Answering a further "layman" question it explained:
'In the context of transformer architectures, a "transformer layer" typically refers to a single self-attention mechanism layer, while a "transformer block" consists of multiple layers stacked together, often including not only self-attention layers but also feedforward layers and normalization layers.'

A further question about self-attention layers got me an example explained (abbrev.):
'how attention weights might be assigned to different parts of the input sequence "why is snow white?"
1. Initialization: each word ("why," "is," "snow," "white") is associated with a query, key, and value vector.
2. Self-Attention Calculation [..] The attention scores are determined by the compatibility (dot product) between the query vector of the current word and the key vectors of all other words.
3. Weight Calculation [After using the softmax function to normalize] high attention weights indicate that the current word should focus more on certain words in the sequence.
4. The final representation of each word is obtained by taking a weighted sum of the value vectors, where the weights are the attention weights.

[..]

  • If the model is focusing on "snow," it might assign higher attention weights to "white" to capture the relationship between the two.
  • The attention weights for each word are dynamically determined based on the context and the model's learned patterns during training.'

Next I actually asked "Why is snow white?" and got:
'Snow appears white because it reflects and scatters all the colors of light in the visible spectrum, creating the perception of a white color.'

Asking slightly different "Who is snow white?" I got:
'Snow White is a fictional character and the protagonist of a German fairy tale, famously known from Disney's adaptation.'

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.