Effectiveness of tf-idf on documents with repeated keywords

Question

I was doing some ML reading and came upon tf-idf. The tf portion counts the relative frequency of a relevant word in a document, while idf measures how common or rare a word is across the corpus.

The IDF formula states that IDF = -log(n_t / N), with n_t being the number of documents with the word and N being the corpus. My understanding is that IDF reduces the weights of filler words, such as 'a' and 'the', to 0.

However, how does the calculation work if all the documents have a relevant keyword? For example, the keyword 'healthcare' may appear in a corpus of presidential speeches. But because the word appears in all the speeches, its idf is 0 and hence the tf-idf is 0, effectively assigning a keyword with a weight of 0.

Is there a variant of TFIDF that tackles this issue?

Answer 1

First, please note that TFIDF is a very simplistic weighting method.

The principle of the IDF part is indeed to lower the score of words which appear in many documents and give rare words more weight. The rationale is that rare words are more discriminative.

In classification a "keyword" which appears in all the documents (or even most of them) cannot be relevant because knowing that the keyword appears doesn't contribute to finding the class of the document. The corpus on which the IDF is calculated is supposed to be a representative sample for whatever task one intends to do. If 'healthcare' belongs to every document in this corpus, then it is assumed that healthcare always belong to any document and therefore doesn't bring any semantic information about the document, similarly to a stop word. It would be pointless to use this word as a feature.