4
$\begingroup$

I have a collection of documents, where each document is rapidly growing with time. The task is to find similar documents at any fixed time. I have two potential approaches:

  1. A vector embedding (word2vec, GloVe or fasttext), averaging over word vectors in a document, and using cosine similarity.
  2. tf-idf or its variations such as BM25.

Will one of these yield a significantly better result? Has someone done a quantitative comparison of tf-idf versus averaging word2vec for document similarity?

Is there another approach, that allows to dynamically refine the document's vectors as more text is added?

Improve this question
$\endgroup$
2
  • $\begingroup$ Have you considered shingling? $\endgroup$
    – D.W.
    Apr 18, 2017 at 22:53
  • 1
    $\begingroup$ There are many approaches for document similarity matching, depends on your preference over efficiency or accuracy. SemEval has been running Textual Similarity tasks for several years now, so you can check it out if you're interested. You might want to try LSH if efficiency is preferred (especially when streaming) and you have a large set of data. There is also Word Movers Distance that is very interesting, but practically speaking very slow I afraid w/o relaxation on its criterion.. $\endgroup$
    – Blue482
    Jun 17, 2017 at 19:53

2 Answers 2

Reset to default
2
$\begingroup$

Here some points on which we can focus -

1.) Averaging the words vectors lose the order of words, making it very similar to the concept of Bag of Words. That is why with less Data Bag of words approach perform better than word2vec. it has been found that training word2vec with potentially large data , outperform Bag of Words models. Google's results are based on word vectors that were learned out of more than a billion-word corpus.

2.) Published literature, distributed word vector techniques have been shown to outperform Bag of Words models. In this paper, an algorithm called Paragraph Vector is used on the IMDB dataset to produce some of the most state-of-the-art results to date.

3.) In your case Document size increase with time. So a good approach would be not to cluster each word rather than clustering documents. Word2vec algorithm has power to learn embedding for each document , which we call Doc2vec. it will cluster similar documents. and using cosine similarity you can find similar documents.

Improve this answer
$\endgroup$
0
$\begingroup$

I have few documents(about 71). So far, TF-IDF is the best way because my each documents are not that long.

Plus, for document, doc2vec come from word2vec is better. The reason is doc2vec has more vectors represent document (not words vector).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.