How to get word embedding in CBOW?

Question

I find clear explanations for skip-gram model. We take the output weight matrix, multiply it with the one-hot vector of the word we want to get the embedding.

How does it work in case of CBOW?

I know we have to take one of the input weights (Wvnx) but which one?

Answer 1

CBOW while training tries to predict the main word from the context words.

Once the training is done we use the weight matrix of first hidden layer to get Word2vec embedding. The weight matrix is multiplies by the one hot encoding to get the word2vec emebdding.

If you multiply a 1 x 10,000 one-hot vector by a 10,000 x 300 matrix, it will effectively just select the matrix row corresponding to the “1”. Here’s a small example to give you a visual. :

This means that the hidden layer of this model is really just operating as a lookup table. The output of the hidden layer is just the “word vector” for the input word.

References : http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/

Answer 2

Word2Vec

---

Word2Vec is a predictive model for learning word embeddings from raw text. It takes as its input a large corpus of words and produces a vector space, typically of several hundred dimensions, with each unique word in the corpus being assigned a corresponding vector in the space.

Word vectors are positioned in the vector space such that words that share common contexts in the corpus are located in close proximity to one another in the space. Gensim provides a great way to use and start with Word2Vec.

Word2vec is actually a collection of two different methods:

• Continuous Bag-of-Words (CBOW)
• Skip-Gram

Given a word in a sentence, let's call it $w(t)$ (also called the center word or target word), CBOW uses the context of surrounding words as input. For instance, if the context window $C$ is set to $C=5$, then the input would be words at positions $w(t-2)$, $w(t-1)$, $w(t+1)$, and $w(t+2)$. Basically the two words before and after the center word $w(t)$. Given this information, CBOW then tries to predict the target word. The objective function for CBOW is:

\begin{align}J_\theta = \frac{1}{T}\sum^{T}_{t=1}\log{p}\left(w_{t}\mid{w}_{t-n},\ldots,w_{t-1}, w_{t+1},\ldots,w_{t+n}\right)\end{align}

word2vec CBOW and skip-gram network architectures

Skip-Gram model

• Predicts source context words from the target words.
• Skip-gram treats each context-target pair as a new observation, and this tends to do better when we have larger datasets.

CBOW - Continuous Bag-of-Words model

• Predicts target words from source context words.
• CBOW smoothes over a lot of the distributional information (by treating an entire context as one observation). For the most part, this turns out to be a useful thing for smaller datasets.

More specifically, we use the one-hot encoding of the input word and measure the output error compared to the one-hot encoding of the target word. In the process of predicting the target word, we learn the vector representation of the target word.

• How does Word2Vec produce word embeddings?