0
$\begingroup$

Let's take the CBOW (continuous bag of words) model as the example.

Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input vector is $v \cdot c$, where $v$ is the total number of words to be learned.

Suppose the first hidden layer has $d$ neurons, which is the size of the embedding dimension.

So all weights (or: embedding) parameters that are to be learned can be arranged in a $v \times d$ matrix.

My question is concerning how many weight parameters are updated during each training epoch for the word2vec model. Are all the $v \cdot d$ embedding, parameters updated during each training epoch? Or, are only $c \cdot d$ of them updated during each training epoch? Here the $c \cdot d$ embedding parameters are for the input context words.

If all $v \cdot d$ weights got updated during each epoch, it would be much more expensive, considering that the typical value of $v$ ranges from tens of thousands to hundreds of thousands.

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.