The authors of the Paragraph Vector paper describe PV-DBOW with:
2.3. Paragraph Vector without word ordering: Distributed bag of words
The above method considers the concatenation of the paragraph vector with the word vectors to predict the next word in a text window. Another way is to ignore the context words in the input, but force the model to predict words randomly sampled from the paragraph in the output. In reality, what this means is that at each iteration of stochastic gradient descent, we sample a text window, then sample a random word from the text window and form a classification task given the Paragraph Vector.
I have a couple of questions:
- Why do you need to sample a text window before sampling
a random word? To create a batch, why can't you just randomly sample from a list of the form
[(1, "cat"), (1, "sat"), ..., (1, "mat"), (2, "humpty"), (2, "dumpty"), ... (2, "wall"), ...]where the first item in each tuple represents the paragraph?
- If hierarchical softmax or negative sampling is used, is stochastic gradient descent still used to update the weights in the network? Or are these optimization methods themselves?
- To infer representations for new paragraphs, is the model only trained on words sampled from that paragraph?