3
$\begingroup$

I'm wondering how Word2Vec is being constructed.

I've read tutorials simply stating that we can train a skip grams neural network model and use the weights that are trained as word vectors.

However, I've also seen this picture:

enter image description here

If reading this diagram correctly:

1) The CBOW and Skip Grams model are both trained with some inputs

2) The output of CBOW is used as an input to the middle neural network

3) The output of skip grams is used as an output to the middle neural network.

The output of CBOW is the a prediction of the center word given a context, and the output of skip grams is the prediction of the surrounding center word.

These outputs are then used to train another set of neural network.

Hence we first train the CBOW, and Skip-gram neural network, then train the middle neural network afterwards? And the input to the middle neural network is one hot encoded.

Is the above interpretation correct?

$\endgroup$
1
$\begingroup$

There is only one neural network to train in word2vec. CBOW, continuous-bag-of-words', and skip-gram are two different methods of constructing the training data for the neural network. You have to pick one of those two training methods.

| improve this answer | |
$\endgroup$
0
$\begingroup$

To understand exactly how word2vec model is defined and trained it may be helpful to look at code based on this tutorial:

https://github.com/tensorflow/tensorflow/blob/r1.3/tensorflow/examples/tutorials/word2vec/word2vec_basic.py

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.