Word Frequency Analysis of Document Sets

Question

I'm doing some work trying to extract commonly occurring words from a set of human classified documents and had a couple questions for anyone who might know something about NLP or statistical analysis of text.

We have a set of a bunch of documents, and users have classified them as either good or bad. What I'd like to do is figure out what words are common to the good documents, but not necessarily the other ones.

I could, for example, use the (frequency within good documents / total frequency) which would essentially normalize the effect of a word being generally common. This, unfortunately, gives very high precedence to words that occur in only a few good documents & not at all in the other documents. I could add some kind of minimum threshold for # of occurrences in good docs before evaluating the total frequency, but it seems kind of hacky.

Does anyone know what the best practice equation or model to use in this case is? I've done a lot of searching and found a lot of references to TF-IDF but that seems more applicable for assessing the value of a term on a single document against the whole set of docs. Here I'm dealing with a set of docs that is a subset of the larger collection.

In other words, I'd like to identify which words are uniquely or more important to the class of good documents.

Answer 1

There are many algorithm to do classification: Naïve Bayes, logistic regression, SVM, decision tree..etc. My suggestion is to try Naïve Bayes first by calculating below probabilities which a new document belongs to $class_{good}$ or $class_{bad}$. (https://web.stanford.edu/class/cs124/lec/naivebayes.pdf)

$$ P(Class_{good} \vert document_{new}) = \frac{P(document_{new} \vert Class_{good}) \cdot P(Class_{good}) }{P(document_{new})} $$

$$ P(Class_{bad} \vert document_{new}) = \frac{P(document_{new} \vert Class_{bad}) \cdot P(Class_{bad}) }{P(document_{new})} $$

And generally, when we are doing text mining questions, we will do several preprocess on one document:

• Tokenization(1-gram/bigram/...etc)
• remove stop words ('a', 'the' 'at', ... etc)
• Stemming: transforms a word into its root form. (studied => study)

My suggestion is to do above preprocesses and try more features not just the words in one document, if there are some metadataes.

Answer 2

I guess what you are looking for is Differential Word Usage. This method takes two text corpora as input and you can get the list of words which are being used more in one text corpus over the other.

Basically what you need to do is to build a common Term Document Matrix for the corpora you are using and then divide this TDM into two TDMs such that all the document columns from corpus 1 fall in one TDM and all the documents columns form corpus 2 fall in the second TDM. For example, you have 2 corpora, the first one containing 10 documents and the second one containing 15 documents. You first, combine both these corpora and form 25 document corpus and then form the TDM, where terms become the rows (let's say there are 300 terms) and the 25 documents become the 25 columns. Here the first 10 columns represent the documents of first corpus and the remaining 15 belong to the second corpus. So, you divide this TDM of dimensions 300 x 25 to two TDMs of dimensions 300 x 10 and 300 x 15. Then you can use Chi-square difference over these TDMs to determine which words are occurring more in one corpus than the other.

A wonderful example has been given regarding this approach by Vik in his blog using Wikileaks corpus and R here: http://www.vikparuchuri.com/blog/finding-word-use-patterns-in-wikileaks/

Answer 3

It sounds to me like you have a binary classification problem (classifying good vs. bad documents for some definitions of good and bad) and the words are being used as features or "signals" for what predicts good vs. bad documents. One thing you might try is to measure some type of correlation statistic between unigrams and each class you're interested in. This preserves your requirement of measuring occurrences of words given a target class over groups of documents.

So, to be a bit more concrete, you could split your documents into two sets (good and bad), and then tokenize your documents to obtain individual terms. From here you could really choose whichever term weighting scheme you like (TF, TF normalized against the length of the document, TF-IDF) and measure your correlation statistic between all these unigrams and the class of interest. You could then produce a ranking based on the correlation coefficients for each term, and take the top-k terms. Some correlation statistics you might try could be Chi-squared (which would measure "lack of independence" between terms and a class). There's also a nice implementation of Chi-squared test for feature selection in Python's Scikit-Learn machine learning library that may be a place to start for this task. Hopefully, that helps!

Answer 4

Your (frequency within good documents / total frequency) seems reasonable to me. It could be that most words that occur in many good documents, simply also occur in many bad documents.

How about you make a list of all the words appearing in the good documents. Then you count their appearance in the good documents and their appearance in the bad documents and compare those two numbers. The words that appear more often in the good ones, with a difference higher than a certain threshold are the ones of interest to you (if they exist).