Let's take the CBOW (continuous bag of words) model as the example.

Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input vector is $v \cdot c$, where $v$ is the total number of words to be learned.

Suppose the first hidden layer has $d$ neurons, which is the size of the embedding dimension.

So all weights (or: embedding) parameters that are to be learned can be arranged in a $v \times d$ matrix.

My question is concerning how many weight parameters are updated during each training epoch for the word2vec model. Are all the $v \cdot d$ embedding, parameters updated during each training epoch? Or, are only $c \cdot d$ of them updated during each training epoch? Here the $c \cdot d$ embedding parameters are for the input context words.

If all $v \cdot d$ weights got updated during each epoch, it would be much more expensive, considering that the typical value of $v$ ranges from tens of thousands to hundreds of thousands.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.