automatic feature selection

Question

I have a lot (thousands are possible) of automatically-generated ordinal features that i'd like to exploit , to differentiate between two classes. I'm looking for some measure that will select the most-likely-to-be-useful ones . For instance I could look at the frequency of occurrence of each feature for one class vs. the other class, and go for those features that have the greatest delta .
If the features are ngrams from given names, and the classes are latin-american vs. american names , the frequency histograms look like this : latin ngrams, english ngrams

Is there some better way to go about this than looking for large frequency deltas? The scikit-learn classifiers allow you to check for feature usefulness post-facto, I was looking to do this pre-classifier.

Answer 1

Frequency delta is not necessary a sound method. A better approach would be to use a statistical test for independence, like chi square independence test which can tell you if two categorical samples does not come from the same distribution. A similar construction can be done with G statistic. See more on wikipedia.

One problem you will certainly encounter is that some categorical distribution for ngrams have a large number of categories and the test is heavily applicable. To avoid that you can group the frequencies which are very small in both distributions into a single category: other. That way you limit the number of categories but comparison remains significant.

Simply comparing differences is not enough since you do not have some threshold to use which have also a meaning, while a statistical test produces p-values which are probabilities in the end.

On the other hand, you should consider the following issue with both approaches (the one you proposed and the one I presented you): they test the individual contribution of one feature, independent of others. As such, they will not catch situations where two features have insignificant contribution, but together they have a significant one.