doc2vec - How does the inference step work in PV-DBOW

Question

I am quite confused about how we generate new paragraph vectors in PV-DBOW?

If I want to use the embeddings to classify some text how would I generate a vector for a new paragraph?

In the original paper the authors wrote:

At prediction time, one needs to perform an inference step to compute the paragraph vector for a new paragraph. This is also obtained by gradient descent.

It is unclear to me how they would compute the paragraph vector for a new paragraph using gradient descent.

Answer 1

The original paper does a lot of hand-waving on the implementation of inference step and is not clear. So your confusion is justified. I'll explain at high level below. I'm assuming only PV-DBOW model.

Training Phase

1. In this model, we forget word ordering information and setup a very simple neural network.
2. Represent all input document tags in a vocabulary to get unique ID for each document tag. Do the same for words.
3. For a given document ID, the input is one-hot encoded representation of document tag IDs. Output is one hot encoded representation of a randomly selected word.
4. We want to setup training such that for a given tag ID, a randomly selected word from that document will be predicted with high probability.
5. So the neural network getting trained transforms one hot encoded representation to document/tag vector. The tag vector is passed through another layer with softmax at the output. Both set of weights are adjusted during training phase.
6. What training achieves: It represents all document tags in a new space such that probabilities for randomly selected words in each document are maximized starting from that vector space representation to softmax output.
7. Important to note that there are two set of weights. Input to hidden layer and hidden to softmax output.

Inference Stage

1. Words are unique across document. So a word 'India' in training and inference stage gets mapped to same ID in vocabulary.
2. But there's not such concept for document ID since documents are assumed to be unique and there's no shared ID between training and inference.
3. Inference stage runs a (sort of) reverse calculation. What vector space representation is most appropriate for this document, if I use the same set of weights from hidden space to output layer? The weights from hidden layer to softmax output are kept constant!
4. Select a word from new document at random. Start with a random representation for the document vector (input to hidden layer). Pass it through neural network from hidden to softmax output with constant weights (learnt during training). Adjust the randomly initialized weights such that softmax probability is maximized for the selected word. Repeat this process many times.
5. This is the reason why you need stochastic gradient descent and need to specify number of steps.

Summary

1. Document IDs are not common from training and inference stage (like word IDs in w2v).
2. There are two set of weights, keep one constants during inference stage based on learning from training phase. Adjust the other using SGD.
3. You mapping new document to document vector space such that documents which had similar words during the training phrase are close in the document vector space.

This is a very good tutorial on the implementation by Richard Berendsen. I have ignored additional complexities like negative sampling, context window from the explanation.

Answer 2

hssay answer is well-written but I thought I add my own explanation to simplify it even more and shows how it works for PV-DM also

Doc2Vec, unlike Word2Vec, is built in a unique way that makes the prediction (or inference as they call it) of the same document slightly different each time. The reason for this slightly un-stable prediction is that the prediction operation actually tries to adjust some weights of the neurons inside the network each time to be able to “correctly” predict the outcome. You might find it odd first but this is why the algorithm is able to predict embedding for documents not only words.

Explanation

Distributed memory model architecture

Before the training starts, we assign a Document ID to each document in our training dataset. So each data point has 2 components: 1) Document ID 2) A Document (list of words.)

First, we transform each word in the corpus to a vector using the traditional Word2Vec algorithm.

Second, we train a model by feeding the network 2 things: 1) Document ID and 2) Context from that Document. The objective of the training phase is to adjust the weights so that for any Document ID and a Context (fixed-length and sampled from a sliding window over the Document), the model can predict the next word with high probability. To do this, we form 2 layers, one to transform the Document ID into vector (Layer A), and another hidden layer (Layer B) with softmax at the output that takes as an input the average (or concatenation) of :

1- The output of Layer A (Vector of Document ID)

2- The embedding of the words in the Contexts done by Word2Vec

Softmax layer outputs the vector representation of the Document. The model trains until all weights are setup in a way to achieves the highest prediction probabilities (or as close it can get).

Now comes the prediction phase. The data point for prediction is only the Document we want to predict its vector (we don’t have Document ID as before). We need Document ID as input for Layer A otherwise we won't’ be able to use the network, so what do we do? We try to predict the Document ID and here is why it’s not deterministic :)

To be able to predict the Document ID:

1- we freeze the hidden layer weights we learned from training (Layer B)

2- We give the network a Context from that Document we want to predict its vector.

3- We use our Word2Vec model to get the embeddings for each word in context

4- We come up a random vector

5- We take the average (or concatenation) of vectors in step 3 and 4 above and use it as input to the hidden layer (which has weights frozen)

4- We check if the random vector we chose did maximize softmax probability for the predicted next word in Context.

5- We repeat this process by stochastically gradient descending on Document ID vector until we find the Document ID vector represents that maximizes the probability for the selected word

6- We use the predicted Document ID along with Document words vectors to predict Document vector representation (which will be as good as the predicted Document ID)

This is why we can't always get the exact same value for the Document vector.

This is how a distributed memory model architecture works, Distributed bag of words also rely on predicting the Document ID to predict the Documents vector but has a different algorithm (explained below if you are interested)

Distributed bag of words architecture:

At the training phase, we train the model by feeding the network 2 things: 1) Document ID and 2) randomly selected words from that Document. The objective of the training phase is to adjust the weights so that for any Document ID, the model can predict the randomly selected words from that document with high probability. To do this, we form 2 layers, one to transform the Document ID into vector (Layer A), and hidden layer (Layer B) with softmax at the output that takes as an input the output of Layer A (Vector of Document ID). Softmax layer outputs the vector representation of the Document.

The model trains until all weights are setup in a way to achieves that (or as close it gets).

For prediction:

1- we freeze the hidden layer weights (Layer B) we learned from training

2- We give the network randomly selected words from that Document we want to predict its vector.

3- We come up a random vector as an input to the hidden layer (Layer B)

4- We check if the random vector we chose did maximize softmax probability for the selected words.

5- We repeat this process by stochastically gradient descending on Document ID vector until we find the Document ID vector represents that maximizes the probability for the selected words

6- We use the predicted Document ID as in input to the hidden layer to predict Document vector representation (which will be as good as the predicted Document ID)