1
$\begingroup$

I'm facing a text classification problem where the algo is human-made but impacted by keywords. Hence I can't use any ML model but I certainly can take a data-science driven approach to find the best keywords that would determine a class.

Therefore I managed to build a dataframe with Countvectorizer, where each row is a word of the whole input data vocabulary, and the 3 categories columns hold the amount of times the word appeared in each category.

What's my best shot at getting the best keywords from that dataframe? Can I use statistical tests for significance ? I can't just choose an arbitrary threshold and say all words having more than 20 occurrences in a category are good keywords for that category; this because the dataset is also quite unbalanced (50% is class A, 35% is class B and 15% is class C) so I'm sure there must be some clever way to extract the most significant impacting keywords.

My first try that somewhat works is weighting occurrences by class occurrences, so having a word appear 15+X% in class C could be significant although the word appears way more often in class A when looking at absolute numbers. This method is not perfect but that's where I'm stuck. Ideas are very much welcome.

$\endgroup$

1 Answer 1

1
$\begingroup$

Can I use statistical tests for significance ? I can't just choose an arbitrary threshold and say all words having more than 20 occurrences in a category are good keywords for that category

It's not just a matter of threshold, because there are word which are frequent in general even though they are not especially associated to a category.

First, in any case you should remove the least frequent words because they often appear by chance and cause overfitting.

The first very basic option is to calculate the conditional probability of a category given a word $p(c|w)$: words which have a high $p(c|w)$ are highly associated with $c$, but this doesn't take into account the imbalance between the categories so you'd have a lot of words which are highly associated with the majority category.

A more proper method is to use a statistical association measure:

$\endgroup$
2
  • $\begingroup$ Very interesting, the PMI is very logical in fact, I will also try conditional entropy. By any chance do you know if those are implemented in known packages like scikit-learn or another ? $\endgroup$
    – Waroulolz
    Mar 11, 2022 at 23:35
  • $\begingroup$ @Waroulolz in sklearn I think the closest is to use the provided feature selection metrics, but they are not exactly the same as the ones I mentioned. See also this question about obtaining the real PMI from the sklearn library (I didn't try it). Note that PMI or info gain are reasonably simple to implement, they just require the probabilities of the two elements, in your case a word and a categ. $\endgroup$
    – Erwan
    Mar 12, 2022 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.