Question regarding training data in word2vec - skip-gram

Question

I have a very simple question regarding the training data in word2vec. In the skip-gram implementation, the training data (if I understand it correctly) is generated as pairs of words like it's shown in this image:

This essentially is just pairs of one-hot vectors. My question is what happens to the results if, instead of splitting every window with one sample per pair of words, I train with the following data: (word, {words in window})? In other words, with a window like the previous one, I would be sending the vector (1, 0, 0, 0, ... 0) to (0, 1, 1, 0, ... 0)/num_non_zero?

(The num_non_zero division is a way to normalize the probabilities so that the softmax layer fits correctly)?

I know I can go and try myself, but I was just wondering if you can shed some light on what results I should expect. Mostly because this is a very obvious alternative to the original and I would be surprised if people haven't used this before.

Answer 1

Word2vec uses a sigmoid output layer, so (unlike in softmax) each dimension of the output (corresponding to the second word in the pair) is treated completely separately. So ``putting through two words at the same time'' doesn't make much sense. If you completely changed things and use softmax, so that you can put multiple outputs through together, then it will be much slower for a start (see original word2vec papers for why/how they avoid softmax).

[But I may be misinterpreting what you're suggesting. Sometimes a good way to get a better feel for things is just to try something that seems to make sense and figure out what happens...]