Continous bag of words claimed to be unsupervised, how is it working?

Question

I'm following these two lectures on CBOW and skip-gram word2vec models. The first is lec 12 and the next lec 13 of a deep learning series

https://www.youtube.com/watch?v=syWB-YMYZvI

https://www.youtube.com/watch?v=GMCwS7tS5ZM&t=548s

Up to about 17 mins into the second video the lecturer says that the approach is unsupervised for CBOW as there are no labels? How can you learn a NN with no labels? This completely confused me as why are we not comparing our softmax probability vector to an actual set of outputs so that we can adjust the v_c and v_w weights accordingly. His liklihood function seems to only be concerned with the parameters v_c and v_w (and completely devoid of some kind of target label) which is bonkers to me, because could i not not just make them whatever i want? In addition how are you learning relationships between pairs of actual words, if actual target variables are not guiding you towards correct labels? Could someone please explain what is going on under the hood as i really want to understand this approach. Perhaps target labels are being considered and i've not noticed it?

Most log likelihood estimates with a window of size $m$ look something like the following $-\mathrm{log} \prod_{j=0,j \neq m}^{2m} \frac{e^{u^T_{c-m+j}v_c}}{\sum_{k=1}^{|v|} e^{u_k^Tv_c}} $

To my knowledge a likelihood function should involve data from actual observations not purely parameters, which are to be estimate, can someone explain this also?

Note i follow it more from 17 mins onwards as he talks about pairs (w,c) in D or D' respectively. I appreciate any help!

Answer 1

The CBOW approach is unsupervised because the network learns the distribution of word co-occurrences around each word, and this doesn't require labelling or additional input, just sequences of words.

As Mikolov et al state in one of the original articles, "the training objective is to learn word vector representations that are good at predicting the nearby words; in another, Mikolov et al say their aim is "building a log-linear classifier with four future and four history words at the input, where the training criterion is to correctly classify the current (middle) word". So if you can see that for each sequence, you're taking 'x' words and training the network to predict one given the others, then no supervision is involved.

It's a little unconventional because it's not the output of the network which is important, but the weights that are learned during training - these are what is taken and used as embeddings in other tasks.

Adrian Colyer does a great general write-up of word2vec here, and the explanation from Chris McCormick here is good and quite accessible.

Answer 2

It is considered "unsupervised" since labels do not have to be created manually.

Labels are generated automatically based on how words are used in real world.