4
$\begingroup$

Which word embedding mechanism does chatGPT use? Is it Word2Vec, GloVe, or something else?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Something else. The answer is well-written on its website. Please do your own research next time. $\endgroup$
    – lpounng
    Commented Jan 27, 2023 at 3:39

1 Answer 1

Reset to default
4
$\begingroup$

ChatGPT is a language model based on the Transformer neural architecture, but only the decoder part.

It does not use pre-trained embeddings to represent text. Instead, the embeddings its uses are trained with the rest of the neural network. Also, the granularity ChatGPT uses to represent text is subwords, not whole words.

Its predecessor, GPT-2 is open source, so you can get an idea of ChatGPT's internals by having a look at it. GPT-3 is a "scaled up" version of GPT-2, both regarding the model and the training data. The jump from GPT-3 to GPT-3.5 (text-davinci-003) and ChatGPT is more involved than just scaling; check this article out for details.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thank you very much. That article is too hard for me to read not only due to my limited English reading skills but also the professionality of that article. This is the first time I heard of subwords, I will search for it to learn more. $\endgroup$
    – zzzgoo
    Commented Jan 28, 2023 at 11:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.