I am dealing with a data where I have only two documents and there are some words which are present in both. Now Term Frequencies (tf) of these words are very high for respective single document than the other. For e.g.

Word1 is present in Documents D1 and D2, and
tf(Word1,D1) = 1000
tf(Word1,D2) = 3
But since Word1 is present in both the documents 
IDF(Word1) = 0
TF-IDF(Word1,d) = 0 for all d belonging to {D1,D2}

So inspite having a very powerful presence in a single document, TF-IDF score will always be 0. One solution I could think is to take word1 as absent if tf(Word1) < threshold. However I still don't feel this is good enough as the Word present in only one single document is given IDF score of only 0.5. I feel like TF-IDF is not a good measurement when number of documents are very low. Any suggestions here?


Weighting scheme 2 in table Recommended TF-IDF weighting schemes in tf–idf | Wikipedia should solve your problem.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.