I've been reading about the Skipgram model and I have found what I interpreted as multiple definitions.
1 - Taking a look at this blog post and Andrew Ng's Deep Learning Specialization, I understood that, for each word, we generate one training sample for each context word.
So, if we have the sentence "cat sat on the mat" we will have samples:
(cat, sat)
(sat, cat)
(sat, on)
And so on.
Then you train your network which will have the dimensions.
- Input: $(V,1)$
- Weights 1: $(d,V)$
- Hidden layer: $(d,1)$
- Weights 2: $(V,d)$
- Output layer: $(V,1)$
Ok, so these dimensions match and we are good. For one given input, given that the weights are the same, we always have the same estimation in the output layer.
In this definition we have symmetric samples (for example, (cat, sat)
and (sat, cat)
) and saying that we use center words as inputs and context words as outputs is meaningless since they are interchangeable?
2 - Watching Stanford's NLP with Deep Learning class (38:54), it seems like for the same center word we can get different outputs:
The numbers in the red circles that I drew should be the same from what I understand. I don't really understand how it is possible to get different outputs if you multiple Weights 2 by the Hidden Layer.
3- I saw in other places (couldn't find reference now) another formulation: for each input word, we represent the context as a vector with ones for the context words and zeros elsewhere. So, in this case, instead of one training example for each context word, we would have one training example for each center word.