I need to find cosine similarity between two text documents. I need embeddings that reflect order of the word sequence, so I don't plan to use document vectors built with bag of words or TF/IDF. Ideally I would use pre-trained document embeddings such as doc2vec from Gensim. How to map new documents to pre-trained embeddings ?

Otherwise what would be the easiest way to generate document embeddings in Keras/Tensorflow or Pytorch?


There are several ways you can obtain document embeddings. If you want to obtain a vector of a document that is not part of the trained doc2vec model, gensim provides a method called infer_vector which allows to you map embeddings.

You can also use bert-as-service to generate sentence level embeddings. I would recommend using Google's Universal Sentence Encoder (USE) model to generate sentence embeddings if your goal is to find some sort of similarity between sentences or documents. There are ways you can combine sentence level embeddings to document level, the first step to try would be to take the mean, or you could generate sentence embeddings for a sliding window over the document and take the mean of that.

The reason I recommend USE over BERT is that, USE was trained specially for sentence similarity tasks whereas BERT, even though can be applied to any NLP task was original trained to predict words in a sentence or complete a sentence. You might find this link helpful, it draws a great comparison between USE and BERT, and why it is important to choose a model based on task.

  • $\begingroup$ Nicely written but computing embeddings from a USE model is goons be a costly approach, right? After all it's not so light an arch (same for Bert) (unless not done on GPU) $\endgroup$ – Aditya Mar 15 '20 at 8:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.