I am creating a Doc2Vec model out of hundreds of PDF documents.

I have 17 documents that are part of this Doc2Vec that I want to use to check similarity with other documents in the Doc2Vec model.

For instance I want to something like: model.similarity(tag5, tag30)

Can this be done?

  • Given that you have Doc2Vec already, have you tried visualizing few documents with T-SNE to see if vector alone can segment documents in cluster of meaningful groups? – Shamit Verma Dec 6 at 15:01

Based on what you have shared, it seems like Doc2Vec should be suited to your objective.

That said, I think the Doc in the package name can lead people astray. It gives the impression that you can feed in any documents and it will find the similarities among them. And while you certainly can feed whole documents, the reality is that the similarity / dissimilarity may not be aligned in a useful way to your task. I have found that training the model on smaller chunks of the document, like sentences or paragraphs, allows for the model to identify subtle differences that can then be aggregated to the whole document.

In your case, are the 17 documents you mentioned, exemplars of 17 different types of documents whose label you want to assign to the entire corpus? If so, Doc2Vec may be overkill. Have you looked into using Tf-Idf? Sometimes this approach works well if you don't necessarily need the word embeddings. Just some food for thought.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.