2
$\begingroup$

Does anyone know of reliable sources that have written about the architecture of OpenAI's ChatGPT - specifically regarding the following?:

  • Number of hidden layers

  • Number of attention heads

  • Dimensions of its hidden layers

  • Sequence length in terms of number of tokens

  • Number of parameters in the version of the model currently free for public preview. My understanding is there are multiple versions of ChatGPT.

Please note: I'm familiar with GPT-3 already. However, I am not certain that ChatGPT's architecture is the same. Open to contrary evidence, if any.

Improve this question
$\endgroup$
4
  • $\begingroup$ There is no open-source version of ChatGPT. What are you referring to exactly? $\endgroup$
    – noe
    Commented Feb 3, 2023 at 16:53
  • $\begingroup$ My apologies - I meant the freely accessible version of ChatGGPT for the public to preview. $\endgroup$
    – user141493
    Commented Feb 3, 2023 at 16:54
  • $\begingroup$ Please, consider upvoting and accepting the given answer or, alternatively, describe why you consider they are not correct or not clear enough. $\endgroup$
    – noe
    Commented Feb 6, 2023 at 7:10
  • $\begingroup$ Yes, I will do that. Please also consider upvoting my question should you feel so inclined. $\endgroup$
    – user141493
    Commented Feb 7, 2023 at 4:24

1 Answer 1

Reset to default
2
$\begingroup$

From the OpenAI website, we know that ChatGPT is a fine-tuned version of GPT-3.5 (text-davinci-002).

On the GPT-3.5 presentation page, they mention that the number of parameters is 175B (in the footnote of the table there, we can read that there may be slight differences with the actual model used in the API, though), which matches the size of GPT-3. Therefore I understand that ChatGPT is the same size as GPT-3.

The details of the architecture of GPT-3 have been made public in the paper:

enter image description here

Also, in the paper they mention that:

All models use a context window of nctx = 2048 tokens

So:

  • Number of layers: 96
  • Number of attention heads: 96
  • Dimensions of its hidden layers: 12288
  • Sequence length: 2048
  • Number of parameters: 175B

Note that, as ChatGPT comes from GPT-3 and GPT-3 is an evolution of GPT-2, we also know from their papers that the architecture differs somewhat from a standard Transformer decoder. See this answer for details on the specific architectural differences.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.