I have a very simple question regarding the training data in word2vec. In the skip-gram implementation, the training data (if I understand it correctly) is generated as pairs of words like it's shown in this image:
This essentially is just pairs of one-hot vectors. My question is what happens to the results if, instead of splitting every window with one sample per pair of words, I train with the following data: (word, {words in window})
? In other words, with a window like the previous one, I would be sending the vector (1, 0, 0, 0, ... 0)
to (0, 1, 1, 0, ... 0)/num_non_zero
?
(The num_non_zero
division is a way to normalize the probabilities so that the softmax layer fits correctly)?
I know I can go and try myself, but I was just wondering if you can shed some light on what results I should expect. Mostly because this is a very obvious alternative to the original and I would be surprised if people haven't used this before.