1
$\begingroup$

I have gone through several links but was not able to understand how CBOW and Skip Gram is trained from scratch?

Any good link/blogs or books would be very helpful.

Word2Vec - CBOW and Skip-Grams

What's the difference between Skipgram word2vec and CBOW word2vec during training and when to use CBOW Skip-gram. ?

Example or Application where CBOW would be preferable choice but not Skip-gram and vice versa.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

Here is a example for what @Christos Karatsalos is describing:

If you take a statement like "The cat jumped over the puddle",

In CBOW's case, the neural network's word embeddings are trained by passing a input set of words to the neural network and making the network to predict an output word as shown below:

$\{ cat, jumped, over, the, puddle\} \rightarrow The $

$\{ The, jumped, over, the, puddle\} \rightarrow cat $

$\{ The, cat, over, the, puddle\} \rightarrow jumped $

$\{ cat, jumped, the, puddle\} \rightarrow over $

$\{ The, cat, jumped, over, puddle\} \rightarrow the $

$\{ The, cat, jumped, over, the\} \rightarrow puddle $

In skip-gram's case, the neural network's word embeddings are trained by passing an input word to the neural network and teaching the network to predict a set of words as shown below:

$The \rightarrow \{ cat, jumped, over, the, puddle\} $

$cat \rightarrow \{ The, jumped, over, the, puddle\}$

$jumped \rightarrow \{ The, cat, over, the, puddle\}$

$over \rightarrow \{ cat, jumped, the, puddle\}$

$the \rightarrow \{ The, cat, jumped, over, puddle\}$

$puddle \rightarrow \{ The, cat, jumped, over, the\}$

For usage, if you are thinking of a fill-in-the-blanks kind of problem CBOW Word2Vec would be a suitable vector to use, on the other hand if you have a word and you are trying to come up with a new sentence with it then skip-gram Word2Vec will be useful.

Take a look this lecture notes for more information.

Improve this answer
$\endgroup$
1
$\begingroup$

CBOW model is able to learn to predict the word by the context, which means that it tries to maximize the probability of the target word by looking at the context. On the other hand, the Skip-gram model is designed to predict the context given the word.

Skip-gram model works well with small amount of training data, moreover it represents well even rare words or phrases.

CBOW model is several times faster to train than the Skip-gram model, and achieves slightly better accuracy for the frequent words.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Why Skip-gram model works well with small amount of training data? $\endgroup$
    – MAC
    Oct 13, 2020 at 6:42
  • $\begingroup$ According to the following paper, Skip-gram works well with small amount of the training data. $\endgroup$
    – Christos Karatsalos
    Oct 13, 2020 at 21:06
  • $\begingroup$ Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 3111–3119. $\endgroup$
    – Christos Karatsalos
    Oct 13, 2020 at 21:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.