I have a lot (thousands are possible) of automatically-generated ordinal features that i'd like to exploit , to differentiate between two classes. I'm looking for some measure that will select the most-likely-to-be-useful ones . For instance I could look at the frequency of occurrence of each feature for one class vs. the other class, and go for those features that have the greatest delta .
If the features are ngrams from given names, and the classes are latin-american vs. american names , the frequency histograms look like this : latin ngrams, english ngrams

Is there some better way to go about this than looking for large frequency deltas? The scikit-learn classifiers allow you to check for feature usefulness post-facto, I was looking to do this pre-classifier.


1 Answer 1


Frequency delta is not necessary a sound method. A better approach would be to use a statistical test for independence, like chi square independence test which can tell you if two categorical samples does not come from the same distribution. A similar construction can be done with G statistic. See more on wikipedia.

One problem you will certainly encounter is that some categorical distribution for ngrams have a large number of categories and the test is heavily applicable. To avoid that you can group the frequencies which are very small in both distributions into a single category: other. That way you limit the number of categories but comparison remains significant.

Simply comparing differences is not enough since you do not have some threshold to use which have also a meaning, while a statistical test produces p-values which are probabilities in the end.

On the other hand, you should consider the following issue with both approaches (the one you proposed and the one I presented you): they test the individual contribution of one feature, independent of others. As such, they will not catch situations where two features have insignificant contribution, but together they have a significant one.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.