How OpenAI embeddings work?

Question

I was looking at the Stanford CS224N NLP with Deep Learning lecture, and in the first two videos, we are introduced to word2vec models. The high-level idea mentioned was that we have a 'big corpus' of text, and then we use SG or CBOW to generate the embeddings (There will of course be many more models). While watching the video I was comparing the approach with how we generate embeddings from OpenAI and this raised a few questions for me

1. What is the 'big corpus' of text that OpenAi uses? Is it only my input? If I send only one word like 'cat' it gives an output, so where does the context come from? If OpenAI doesn't use this approach how does it generate the output? And how is it so fast?

2. Are the OpenAI embedding models substitutes for SG or CBOW?

3. How does the vector embedding generation differ from words, compared to that of sentences or paragraphs? Are they entirely different techniques or extensions of the same approach?

Answer 1

OpenAI described their embeddings in this article. In the last version, they also incorporate the matryoshka representation learning technique.

The answers to your questions:

1. The corpus used to train the OpenAI embeddings is kept private by OpenAI, so we don't know anything about it. When you send only a word, that single word is what is encoded into the embedded vector. The output is generated by a neural network. It's fast because they probably use powerful hardware.

2. Skipgrams and Continuous Bag of Words are approaches to get word embeddings, while OpenAI embeddings are text embeddings, they compute a representation for any piece of text (sentence, paragraph), not at word-level. While some use cases of word-embeddings can be addressed better with text embeddings, they are essentially different things.

3. For OpenAI embeddings, individual words are not different to sentences. They are just pieces of text.