1
$\begingroup$

Say I have a corpus of text documents on which I have calculated each documents TfIDF vector. With this sparse matrix representation of the corpus, I can calculate similarities between documents by calculating the cosine similarity between the documents' TfIDF vectors.

If I now compile a set of bigrams based on frequency, and replace instances of two words in each document with the concatenation of the bigram, what affect will this have on

1) Calculating TfIDF vectors?

2) Calculating document similarity?

To be explicit, lets take an example in python:

doc1 = ['the', 'car', 'drove', 'from', 'new','york', 'to', 'washington'] # a single text document
top_n = {('new', 'york'), ('drove', 'from')...} # set of top n bigrams by frequency
new_doc = replace_ngrams(doc1) # function that replaces words with concatenated bigrams
print(new_doc)
>>> ['the', 'car', 'drovefrom', 'newyork', 'to', 'washington']
$\endgroup$
1
$\begingroup$

TFIDF decreases as term frequency will be decreased linearly and idf increases log linearly.

Document similarity will decrease as value of tfidf vectors should decrease as reputation of bigrams are more less than each single word.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.