How to measure the similarity between two text documents?

Question

Assume, I have 100 text documents, and I want to cluster those documents.

The first step is the construct pairwise similarity matrix 100X100 for the documents

My question is:

what are common way to measure similarity between two documents?

Thanks,

Answer 1

In general,there are two ways for finding document-document similarity

TF-IDF approach

1. Make a text corpus containing all words of documents . You have to use tokenisation and stop word removal . NLTK library provides all .
2. Convert the documents into tf-idf vectors .
3. Find the cosine-similarity between them or any new document for similarity measure.

You can use libraries like NLTK , Scikit learn ,Gensim for Tf-Idf implementation . Gensim provides many additional functionality . See : https://www2.cs.duke.edu/courses/spring14/compsci290/assignments/lab02.html

Word Embedding

Google's Doc2Vec ,which is available in Gensim library can be used for document similarity .Additonaly,teh Doc2Vec model itself can compute the similarity score ( no cosine or anything needed her ) . You just need the vectorise the docs by tokenizing ( use NLTK ) and make a Doc2vec model using gensim and fins similarity and many using Gensim inbuilt methods like model.n_similarity for similarity between two documents .

Other

Additionally,since your aim is to cluster documents,you can try Topic Modelling using LDA ( Latent Dirichet Allocation) or LSI ( Latent Semantic Indexing ) .

Answer 2

The most common way is to measure the similarity between two text documents is distance in a vector space. A vector space model can be created by using word count, tf-idf, word embeddings, or document embeddings. Distance is most often measured by cosine similarity. Once all the documents are in the same vector space, the 100x100 matrix with cosine similarity can be calculated.

"Introduction to Information Retrieval" book goes into greater detail.

Answer 3

I would probably approach this by creating a set of features for each document, you could, for example, use a Bag of words representation or TF-IDF term weighting. Following this, you can compute the closeness of these features through some distance metric and use this as a measure of the similarity.