I am working with TF-IDF and cosine similarity to do document comparisons and given a document, which document in the data is the most similar. However, sometimes it returns a high similarity between two documents which don't seem very similar when reading through the document pair. Are there conventional ways of evaluating which words contribute to the high similarity score between two documents using TF-IDF and cosine similarity?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Yes, Cosine TF-IDF is quite transparent so it's usually reasonably easy to visualize the words which contribute the most to a score. Cosine is defined as the dot product divided by the product of the norms, so you can isolate the terms:
dotproduct(d_1,d_2) = tfidf(w1,d1) * tfidf(w1,d2) + tfidf(w2,d1) * tfidf(w2,d2) + ... + tfidf(wN,dN)
Ranking the words w_i by descending order of the term tfidf(w_i,d1) * tfidf(w_i,d2) gives the top most contributing words for the similarity score.
Mind that if the documents have big differences in size, this will have an effect since cosine is normalized with their norms.
