Training Doc2Vec and Word2Vec at the same time

Question

As far as I can tell the typical Doc2Vec implementation (e.g. Gensim) first trains the word vectors and afterwards the document vectors were the word vectors are fixed.

If my goal is that conceptually similar vectors (regardless of whether they represent documents or words) are close to each other, then would it be better to train them simultaneously?

For instance, a document about Godzilla should be similar to the word Godzilla (as well as to other documents about Godzilla) as well as to the word Japan.

Furthermore, could such a result being achieved with a typical implementation, when including single words as documents into the training data?

Answer 1

You need to be careful with the assumptions you make about the doc2vec implementation. Here are some useful concepts:

Word2vec has two different model implementations (Skip-gram and Continuous-bag-of-words) and Doc2vec has analogous PV-DM and CBOW models:

1. Word2vec

   1. Continuous-bag-of-words (CBOW)
   2. Skip-gram

2. Doc2vec (Paragraph Vector)

   1. Distributed Memory (PV-DM)
   2. Distributed Bag-of-words (PV-DBOW)

In skip-gram the inputs are pairs of words generated by a moving window of size n trough the data where we use one word to predict the other, whereas in CBOW the task is predicting a center word from the surrounding words. In the second case the context word vectors are typically averaged.

The Doc2vec extension works analogously. The DBOW model is analog to skip-gram and what it does is calculating the probability of words in a paragraph given a randomly selected word from that paragraph. The DM however, tries to predict the central word based on a context of words and the context paragraph.

As you can see, not necessarily the doc2vec models need to calculate word embeddings first. In fact, in the gensim doc2vec function there is an parameter for the dbow model that says:

"dbow_words (int {1,0}) – If set to 1 trains word-vectors (in skip-gram fashion) simultaneous with DBOW doc-vector training; If 0, only trains doc-vectors (faster)."

Meaning that during the training phase not even the dbow requires producing the word vectors.

There is a great notebook from rare-technologies that gives a lot of insights about how these models are trained.

Regarding your specific questions, you can train a word2vec and doc2vec model and use the results to compare words and paragraphs (look at this paper). But take into account that doc2vec is not training words and then doing some fancy stuff with them. The training process involves calculating paragraphs directly.