Are the embedding values for a particular word using word2vec Skipgram model the weights of the first layer or the softmax output of the function? Does the embedding value change according to the training corpus?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
The word embeddings are the weights of the first layer i.e. the embedding layer and not the softmax output of the function. The embedding values represent a vector which gives the location of the word with respect to other words in a high dimensional vector space. And yes, the embedding values change according to the training corpus. However, if you are using a given language (for example English) and have a large amount of training data the final values of the vectors will turn out to be pretty close even with training corpus of different contexts.
$\begingroup$
$\endgroup$
The output of the softmax is a probability distribution over all words in the vocabulary, so I assume that when you say "or the softmax output of the function", you actually mean the projection matrix from embedding space to logit space before the softmax.
As you can see in equation (2) of the followup paper of the original word2vec article, the embedding matrix and the projection matrix are actually the same:
$ \DeclareMathOperator{\exp}{exp} p(w_O | w_I) = \frac{\exp({v^{\prime}_{w_O}}^T v_{w_I})}{\sum _{w=1}^W \exp({v^{\prime}_{w}}^T v_{w_I})} $
Embedded vectors are a reflection of word co-occurrence statistics of the data they are trained on and therefore they depend totally on the training corpus. For instance, embeddings trained on a news corpus may give totally different vectors than if we used a corpus of children's books, even if both embeddings where defined over the same vocabulary.
