I'm following these two lectures on CBOW and skip-gram word2vec models. The first is lec 12 and the next lec 13 of a deep learning series



Up to about 17 mins into the second video the lecturer says that the approach is unsupervised for CBOW as there are no labels? How can you learn a NN with no labels? This completely confused me as why are we not comparing our softmax probability vector to an actual set of outputs so that we can adjust the v_c and v_w weights accordingly. His liklihood function seems to only be concerned with the parameters v_c and v_w (and completely devoid of some kind of target label) which is bonkers to me, because could i not not just make them whatever i want? In addition how are you learning relationships between pairs of actual words, if actual target variables are not guiding you towards correct labels? Could someone please explain what is going on under the hood as i really want to understand this approach. Perhaps target labels are being considered and i've not noticed it?

Most log likelihood estimates with a window of size $m$ look something like the following $-\mathrm{log} \prod_{j=0,j \neq m}^{2m} \frac{e^{u^T_{c-m+j}v_c}}{\sum_{k=1}^{|v|} e^{u_k^Tv_c}} $

To my knowledge a likelihood function should involve data from actual observations not purely parameters, which are to be estimate, can someone explain this also?

Note i follow it more from 17 mins onwards as he talks about pairs (w,c) in D or D' respectively. I appreciate any help!

up vote 1 down vote accepted

The CBOW approach is unsupervised because the network learns the distribution of word co-occurrences around each word, and this doesn't require labelling or additional input, just sequences of words.

As Mikolov et al state in one of the original articles, "the training objective is to learn word vector representations that are good at predicting the nearby words; in another, Mikolov et al say their aim is "building a log-linear classifier with four future and four history words at the input, where the training criterion is to correctly classify the current (middle) word". So if you can see that for each sequence, you're taking 'x' words and training the network to predict one given the others, then no supervision is involved.

It's a little unconventional because it's not the output of the network which is important, but the weights that are learned during training - these are what is taken and used as embeddings in other tasks.

Adrian Colyer does a great general write-up of word2vec here, and the explanation from Chris McCormick here is good and quite accessible.

It is considered "unsupervised" since labels do not have to be created manually.

Labels are generated automatically based on how words are used in real world.

  • Thanks for the responses, i have added more detail on exactly where i am confused, if you could add details to fill in the gaps that would be awesome! – Iltl Dec 6 at 21:51

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.