6
$\begingroup$

The explanation from the official doc2vec paper "Distributed Representations of Sentences and Documents" for PV-DBOW is as follows:

Another way is to ignore the context words in the input, but force the model to predict words randomly sampled from the paragraph in the output. In reality, what this means is that at each iteration of stochastic gradient descent, we sample a text window, then sample a random word from the text window and form a classification task given the Paragraph Vector.

According to the paper the word vectors are not stored and PV-DBOW is said to work similar to skip gram in word2vec.

Skip-gram is explained in word2vec Parameter Learning. In the skip gram model the word vectors are mapped to the hidden layer. The matrix that performs this mapping is updated during the training. In PV-DBOW the dimension of the hidden layer should be the dimension of one paragraph vector. When I want to multiply the word vector of a sampled example with the paragraph vector they should have the same size. The original representation of a word is of size (vocabulary size x 1). Which mapping is performed to get the right size (paragraph dimension x 1) in the hidden layer. And how is this mapping performed when the word vectors are not stored? I assume that word and paragraph representation should have the same size in the hidden layer because of equation 26 in 2

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

I haven't read the paper you linked, but I have followed a lecture notes by Richard Socher.

So, basically that mapping matrix is called weight matrices. There are two weight matrices W1 and W2 for input and output mapping. Thus for each word, vectors in both the matrices are updated via backpropagation.

To answer your question, a word is represented by one-hot sparse vector which has the size of Vx1 (V is size of vocabulary), with a value of 1 in one of the position. and when this vector is multiplied with the weight matrix W1 with size NxV, the corresponding embedding vector of size Nx1 (N is size of the required embedding vector) is used in the hidden layer.

Document has no one-hot represented vectors. So, basically there is a document matrix D of size Nxd (d is number of documents) where each column represents a document. In other words, the matrices W1 and W2 need the one-hot representation only for mathematical steps, other than that they are representing each word in each column just as the document matrix D.

Improve this answer
$\endgroup$
8
  • $\begingroup$ In the original paper [cs.stanford.edu/~quocle/paragraph_vector.pdf] they write about PV-DBOW: "We only need to store the softmax weights as opposed to both softmax weights and word vectors in the previous model (PV-DM)." Does it mean that 'W1' and 'W2' are initialized randomly and afterwards are not updated. Are they only used to identify the respective word? $\endgroup$
    – саша
    Mar 15, 2016 at 23:21
  • $\begingroup$ Nope. There is no word weight matrices W1 and W2 in DBOW model. The model only learns paragraph vectors. Each paragraph vectors tries to predict the words in context directly. Thus finally only the document matrix is learned. But in gensim, there is a parameter dbow_words, when it is set to 1, it learns word weight matrices W1 and W2 just as the skip-gram model. $\endgroup$
    – chmodsss
    Mar 16, 2016 at 13:03
  • $\begingroup$ And how exactly are the words predicted from a paragraph vector? Or with which vector I have to multiply a paragraph vector to estimate the probability of a word? And which parameters are updated during gradient descent apart from the paragraph vector? $\endgroup$
    – саша
    Mar 17, 2016 at 1:22
  • $\begingroup$ In the first paper, a softmax function is used to convert the dense vector to log-probability vector but then hierarchical softmax function is used. later, negative sampling method is applied to determine it. I suppose only the document matrix which has the paragraph vectors is updated. By mentioning softmax weights, I guess in the paper they have mentioned about the parameters used in the function which is intrinsic. $\endgroup$
    – chmodsss
    Mar 17, 2016 at 1:28
  • $\begingroup$ As I understand softmax there are the softmax paramters and an input that is multiplied with the softmax parameters. Like in this tutorial: [ufldl.stanford.edu/wiki/index.php/Softmax_Regression ], where the thetas are the softmax parameters and x is the input. In case of PV-DBOW is the input to the softmax function the paragraph vector of interest or is this paragraph vector transformed before? $\endgroup$
    – саша
    Mar 17, 2016 at 20:56
0
$\begingroup$

The weight matrix $W_1$ for input (document) and weight matrix $W_2$ for output (words) do not need to be of the same dimension. Say you have $M$ documents and $V$ words, then $W_1$ would be of dimension $M \times H$ and $W_2$ would be of dimension $H \times V$. $H$ is the dimension for embeddings. In other words, as long as the input and output of the neural network shares the same embedding dimension, then the model can work.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.