Skipgram - multiple formulations?

Question

I've been reading about the Skipgram model and I have found what I interpreted as multiple definitions.

1 - Taking a look at this blog post and Andrew Ng's Deep Learning Specialization, I understood that, for each word, we generate one training sample for each context word.

So, if we have the sentence "cat sat on the mat" we will have samples:

(cat, sat) (sat, cat) (sat, on)

And so on.

Then you train your network which will have the dimensions.

• Input: $(V,1)$
• Weights 1: $(d,V)$
• Hidden layer: $(d,1)$
• Weights 2: $(V,d)$
• Output layer: $(V,1)$

Ok, so these dimensions match and we are good. For one given input, given that the weights are the same, we always have the same estimation in the output layer.

In this definition we have symmetric samples (for example, (cat, sat) and (sat, cat)) and saying that we use center words as inputs and context words as outputs is meaningless since they are interchangeable?

2 - Watching Stanford's NLP with Deep Learning class (38:54), it seems like for the same center word we can get different outputs:

The numbers in the red circles that I drew should be the same from what I understand. I don't really understand how it is possible to get different outputs if you multiple Weights 2 by the Hidden Layer.

3- I saw in other places (couldn't find reference now) another formulation: for each input word, we represent the context as a vector with ones for the context words and zeros elsewhere. So, in this case, instead of one training example for each context word, we would have one training example for each center word.

Answer 1

1. Andrew Ng word2vec @1:29. In the video, his choice of window size is more than one. In his example, if "orange" is the center word, the target word could be "juice", "glass", "my", which are all within the window size of 5.

In your example, "cat sat on the mat", if you choose a window size of one, you can get 7 training samples:

(cat, sat),
(sat, on),(sat, cat),
(on, sat), (on, the), 
(the, on), (the, mat)

If this is the entire corpus, p(cat|sat), probability of sat occurs as the center word, cat being the context, is 0.5, p(sat|cat), cat being the center word, sat being the context is 1. The distributed word representation of "cat" and "sat" being the center word should be different.

2. "it seems like for the same center word we can get different outputs:" That's because they are predicting more than one context words.

@31:35 224 2017 lec2, the slide says "predict surrounding words of radius m for every word". That is, the problem is to find a distribution of context words given the center word, so for one center word, the model will have more than one output words (unless you set m to 1 and specify the direction as well).

"I don't really understand how it is possible to get different outputs if you multiple Weights 2 by the Hidden Layer."

@39:37 224 2017 lec2 The professor did a dot product between the matrix output and the word representation. It's also shown on the top of the note: "$u_o ^T v_c$"

Multiply the hidden layer (d * 1) by Weights 2 (V * d) will get a V*1 vector. That's $u_o$, output vector representation. Then we do $u_o ^T v_c$ to get the output for each context word. After that, we apply softmax and compare that value with the context word's one-hot representation ($y$ vs $\hat{y}$), three comparisons in the prof's example.

3. That sounds like the idea of the Continuous Bag of Words model.