4
$\begingroup$

I'm looking to reproduce the doc2vec, i.e. paragraph vector approach by Le & Mikolov.

Is anyone aware of a full script using Tensorflow? In particular, I'm looking for a solution where the paragraph vectors of PV-DM and PV-DBOW are concatenated.

So far, I found:

There seems to be no solution that relates to the original IMDB data, and covers both PV-DM and PV-DBOW. I am aware of a few gensim scripts, but would like to know whether there's a TF solution out there.

$\endgroup$

1 Answer 1

8
$\begingroup$

Implementations

These are all full scripts using Tensorflow, but just using Tensorflow does not ensure quality.

Bad

WangZ's implementation looks complete from a brief glance, although it only offers Adagrad and Stochastic Gradient Descent, two of the least reliable gradient descents. You really want minibatch gradient descent, which will predictably converge close to the local minimum where you initialize your system (think: optimization problems from calculus class). Beyond that, there's nothing really notable about his or her implementation.

Better

Jonathan's implementation is a bit better than WangZ's but worse than the Machine Learning Cookbook solution. He also uses Gensim instead of TensorFlow, which is not bad except from an engineering standpoint (it doesn't scale quite as well). He uses mini-batch gradient descent, but with hierarchical softmax instead of NCE. This is not going to make a huge difference except perhaps increasing training time.

Best

The Machine Learning Cookbook implementation is really just incredible. It follows all of the recommendations from Google's original papers on the topic, even improving on the gradient descent (this uses mini-batch gradient descent). Notably, this also uses NCE (Noise Contrastive Estimation), which I don't typically see, instead of hierarchical softmax to obtain the error for gradient descent, which Google showed produces slightly better results and works faster. A lot of thought clearly went into this, and whoever was behind it knew what s(he) was doing. I would go with this one if I were you.

PV-DM vs PV-DBOW

PV stands for Paragraph Vectors, the model you are talking about. It is also know as doc2vec. It comes in two flavors, CBOW (Continuous Bag of Words), sometimes also called DBOW (Distributed Bag of Words), and DM (Distributed Memory).

These are both distributed models in that each "neuron" contributes to the training task, but nothing has any meaning without the other "neurons" to give it context.

To paraphrase Google's documentation: The CBOW/DBOW model predicts the target word from a bag of context words (e.g. "mat" from "the cat sits on the") while the DM model predicts a skip-gram from a single word (e.g. "sits .. mat" from "cat").

In practice, research shows that the skip-gram/DM model works better if you have a lot of data and the CBOW/DBOW model works better with less. That said, this technique (and deep learning in general) isn't particularly effective without lots of training data.

IMDB

Any of these implementations can be trained on comment data from IMDB. For example, the Machine Learning Cookbook implementation is hard-coded to pull data from a similar dataset . To change where it gets the data from, you would modify the load_movie_data() function in text_helpers.py (same directory in the repository as your link, but a different file) to pull from your preferred source. For instance, you might want to modify it to pull data from the local file-system instead.

$\endgroup$
1
  • $\begingroup$ I believe that the PV-DBOW doc2vec model is analogous tot he skipgram word2vec model and the PV-DM model is analogous to the CBOW word2vec model. See Lau and Baldwin 2016: aclweb.org/anthology/W16-1609. $\endgroup$
    – nedned
    Commented Jan 11, 2019 at 3:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.