Idf values of English words

I'm working on keyword/phrase extraction from a single document. I started by doing term frequency analysis, but this returns words like "new" which aren't very helpful. So I want to penalize the common words and phrases, for which we normally use idf (inverse document frequency). But since it's for a single document, I'm not sure how to do idf analysis.

Is it possible to use tf-idf method with pre-calculated idf values for (all?) words? And are such values available somewhere?

I don't believe that there are any precalculated idf values out there. Inverse Document Frequency (idf) is the inverse of the number of documents in which a particular word appears in your corpus. If you only have one document, I'm afraid that value is simply 1.

However, if you are looking to get rid of words such as the, as, and it which don't carry much meaning, nltk in Python has some useful tools to remove these "stop words" from your document and might help you.

• Thank you! I've already removed the stopwords from nltk, but that's still not enough. nltk lists only 153 stopwords for English. What I'm trying to do is to penalise the common everyday words. I believe this will be helpful in identifying the keywords better. Dec 16, 2017 at 22:08
• I agree that idf is not the correct term here. What I'm looking for is the frequency of each word (or about 10k most frequent words) in everyday usage. Dec 16, 2017 at 22:11
• Or the rank of most frequent words is also enough, from which I can get approx frequency using Zipf's law Dec 16, 2017 at 22:12

The list of 20,000 most common words in English is avaiable here.

By using Zipf's law, we can obtain the probability of these words as below.

Zipf's Law

In the English language, the probability of encountering the rth most common word is given roughly by P(r)=0.1/r for r up to 1000 or so. The law breaks down for less frequent words, since the harmonic series diverges. Pierce's (1980, p. 87) statement that sumP(r)>1 for r=8727 is incorrect. Goetz states the law as follows: The frequency of a word is inversely proportional to its statistical rank r such that

P(r) = 1/(rln(1.78R)),

where R is the number of different words.

These probability values can be used as a substitute for idf.

Thiraputhi is correct that Zipf's law can be used to get a fairly decent set of IDF values from an ordered list of the 20,000 most common words. However, Google's n-grams have been available since 2012 and these contain the data you are looking for, although you have to extract it from their unigrams (i.e. 1-grams) dataset using awk or some other programming language or tool. If you go to the top of the repo mentioned by Thiraputhi, they even kind of allude to these Google n-grams, strangely enough, and they also mention that the files in their repo are derived from Peter Norvig's 1/3 million most frquently used words list. Norvig claims on his site that these come from Google's "Trillion word corpus". This may be the same corpus that Google uses to generate their n-grams, I'm not sure. But Norvig's 1/3 million word list contains a column for the count of the word in the corpus, and this is the column you are looking for for your IDF frequency.

TF-IDF = frequency (word count) of a term in an indivividual document divided by that word's frequency (word count) in the larger corpus, and this latter term is found in column 2 of Norvig's file. It would be superfluous to approximate this column using Zipf's law when you have the original file containing that column available. Here are the links that answer your question: