Word2Vec
Word2Vec is a predictive model for learning word embeddings from raw text. It takes as its input a large corpus of words and produces a vector space, typically of several hundred dimensions, with each unique word in the corpus being assigned a corresponding vector in the space.
Word vectors are positioned in the vector space such that words that share common contexts in the corpus are located in close proximity to one another in the space. Gensim provides a great way to use and start with Word2Vec.
Word2vec is actually a collection of two different methods:
Given a word in a sentence, let's call it $w(t)$ (also called the center word or target word), CBOW uses the context of surrounding words as input. For instance, if the context window $C$ is set to $C=5$, then the input would be words at positions $w(t-2)$, $w(t-1)$, $w(t+1)$, and $w(t+2)$. Basically the two words before and after the center word $w(t)$. Given this information, CBOW then tries to predict the target word. The objective function for CBOW is:
\begin{align}J_\theta = \frac{1}{T}\sum^{T}_{t=1}\log{p}\left(w_{t}\mid{w}_{t-n},\ldots,w_{t-1}, w_{t+1},\ldots,w_{t+n}\right)\end{align}
word2vec CBOW and skip-gram network architectures
Skip-Gram model
- Predicts source context words from the target words.
- Skip-gram treats each context-target pair as a new observation, and this tends to do better when we have larger datasets.
- Predicts target words from source context words.
- CBOW smoothes over a lot of the distributional information (by treating an entire context as one observation). For the most part, this turns out to be a useful thing for smaller datasets.
More specifically, we use the one-hot encoding of the input word and measure the output error compared to the one-hot encoding of the target word. In the process of predicting the target word, we learn the vector representation of the target word.