Find best features for text classification with Countvectorizer Python

Question

I'm facing a text classification problem where the algo is human-made but impacted by keywords. Hence I can't use any ML model but I certainly can take a data-science driven approach to find the best keywords that would determine a class.

Therefore I managed to build a dataframe with Countvectorizer, where each row is a word of the whole input data vocabulary, and the 3 categories columns hold the amount of times the word appeared in each category.

What's my best shot at getting the best keywords from that dataframe? Can I use statistical tests for significance ? I can't just choose an arbitrary threshold and say all words having more than 20 occurrences in a category are good keywords for that category; this because the dataset is also quite unbalanced (50% is class A, 35% is class B and 15% is class C) so I'm sure there must be some clever way to extract the most significant impacting keywords.

My first try that somewhat works is weighting occurrences by class occurrences, so having a word appear 15+X% in class C could be significant although the word appears way more often in class A when looking at absolute numbers. This method is not perfect but that's where I'm stuck. Ideas are very much welcome.

Answer 1

Can I use statistical tests for significance ? I can't just choose an arbitrary threshold and say all words having more than 20 occurrences in a category are good keywords for that category

It's not just a matter of threshold, because there are word which are frequent in general even though they are not especially associated to a category.

First, in any case you should remove the least frequent words because they often appear by chance and cause overfitting.

The first very basic option is to calculate the conditional probability of a category given a word $p(c|w)$: words which have a high $p(c|w)$ are highly associated with $c$, but this doesn't take into account the imbalance between the categories so you'd have a lot of words which are highly associated with the majority category.

A more proper method is to use a statistical association measure:

• Pointwise Mutual Information between a word and a category
• Information Gain or Conditional entropy, which calculate how much more information information one has about the category when one knows the word. These measures are used in decision tree algorithms to select the best features.