1
$\begingroup$

I'm working on keyword/phrase extraction from a single document. I started by doing term frequency analysis, but this returns words like "new" which aren't very helpful. So I want to penalize the common words and phrases, for which we normally use idf (inverse document frequency). But since it's for a single document, I'm not sure how to do idf analysis.

Is it possible to use tf-idf method with pre-calculated idf values for (all?) words? And are such values available somewhere?

$\endgroup$
2
$\begingroup$

I don't believe that there are any precalculated idf values out there. Inverse Document Frequency (idf) is the inverse of the number of documents in which a particular word appears in your corpus. If you only have one document, I'm afraid that value is simply 1.

However, if you are looking to get rid of words such as the, as, and it which don't carry much meaning, nltk in Python has some useful tools to remove these "stop words" from your document and might help you.

Here is a helpful example.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you! I've already removed the stopwords from nltk, but that's still not enough. nltk lists only 153 stopwords for English. What I'm trying to do is to penalise the common everyday words. I believe this will be helpful in identifying the keywords better. $\endgroup$ – Thirupathi Thangavel Dec 16 '17 at 22:08
  • $\begingroup$ I agree that idf is not the correct term here. What I'm looking for is the frequency of each word (or about 10k most frequent words) in everyday usage. $\endgroup$ – Thirupathi Thangavel Dec 16 '17 at 22:11
  • $\begingroup$ Or the rank of most frequent words is also enough, from which I can get approx frequency using Zipf's law $\endgroup$ – Thirupathi Thangavel Dec 16 '17 at 22:12
2
$\begingroup$

The list of 20,000 most common words in English is avaiable here.

By using Zipf's law, we can obtain the probability of these words as below.

Zipf's Law

In the English language, the probability of encountering the rth most common word is given roughly by P(r)=0.1/r for r up to 1000 or so. The law breaks down for less frequent words, since the harmonic series diverges. Pierce's (1980, p. 87) statement that sumP(r)>1 for r=8727 is incorrect. Goetz states the law as follows: The frequency of a word is inversely proportional to its statistical rank r such that

P(r) = 1/(rln(1.78R)),

where R is the number of different words.

These probability values can be used as a substitute for idf.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.