1
$\begingroup$

how it the embedding model trained? Are the embeddings simply extracted from chatGPT4 or are they trained differently from the beginning (pre-training stage)?

$\endgroup$

1 Answer 1

1
$\begingroup$

OpenAI embeddings are not extracted from chatGPT. They are trained independently.

According to the original article OpenAI used to present their embeddings, the model used to obtain the embeddings is a Transformer encoder (like BERT or RoBERTa) trained on the task to distinguish if 2 pieces of text where consecutive in the original source text/code.

Also, as described in their last announcement, their embeddings now are also trained with a technique called "Matrioska representation learning" (see the article) that enables them to have their dimensionality reduced to take less storage space if needed.

$\endgroup$
3
  • $\begingroup$ Amazing, thanks noe. Do you have reference to openAI mentioning what it used for training their first embedding model? $\endgroup$
    – haneulkim
    Mar 23 at 8:24
  • 1
    $\begingroup$ I am not sure what you are referring to with "what it used for training their first embedding model". In any case, the first link I posted is the original article describing their first embeddings, so that is the most appropriate reference for it. You may also find information on their original announcement of their first embeddings on their website. $\endgroup$
    – noe
    Mar 23 at 8:34
  • $\begingroup$ By "what is used for training their first embedding model" it meant what algorithms and loss functions were used to train the model. And since you've already provided me with the paper, I wanted a link to announcement or blog from openAI that references it (specifically says that their model is trained using methods in the paper). Did I clarify it for you? $\endgroup$
    – haneulkim
    Mar 25 at 0:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.