Difference between Word2Vec and contextual embedding

Question

am trying to understand the difference between word embedding and contextual embedding.

below is my understanding, please add if you find any corrections.

word embedding algorithm has global vocabulary (dictionary) of words. when we are performing word2vec then input corpus(unique words) map with the global dictionary and it will return the embeddings.

contextual embedding are used to learn sequence-level semantics by considering the sequence of all words in the documents.

but i don't understand where we considered context in word embedding.

Answer 1

First, some clarifications:

• Non-contextual word embeddings (e.g. word2vec) assign a real-valued vector to each word.
• Contextual embeddings (e.g. BERT-based) assign a vector to each "token". Depending on the specific model to compute the embeddings, we may have word-level tokens or, most frequently, subword-level tokens.

Now, differences between non-contextual and contextual embeddings:

• While the granularity of non-contextual word embeddings is always words, contextual embeddings usually work at subword level. Therefore you cannot use word embeddings the same way as subword embeddings. This is a frequent source of confusion; I suggest you have a look at other answers in this site regarding this issue: this, this and this. If you are using contextual subword embeddings and need to work at word level, you need an extra step to convert your subword vectors into word-level representations (e.g. averaging all subword-vectors of a word).
• Non-contextual word embeddings always assign the same vector to a given word, independently from the rest of the sentence it is in; because of this, we just need a table with the correspondence of word and associated vector to compute word embeddings of any text. Contextual embeddings compute different vectors for the same token, depending on the rest of the sentence and on its order within the sentence; because of this, we need to process the whole sentence with our contextual embedding model (e.g. BERT) to obtain our token embeddings.
• Word embeddings have a finite word vocabulary, which means that if a word is not on the embedding table, we cannot represent it (i.e. out-of-vocabulary (OOV) problem). With subword embeddings we also have a subword table but normally it contains small subwords, including all letters of the alphabet, so unseen words are usually not a problem as long as they have the same script (Latin script, Cyrillic, Chinese characters, etc) as the training data of the embeddings model; you can check this anwer for more detail.