How does SelectKBest() perform feature selection?

SelectKBest(f_classif, k), where k is the number of features to select, is often used for feature selection, however, I am having trouble finding descriptive documentation on how it works. A sample of how this works is below:

model = SelectKBest(f_classif, k)
model.fit_transform(X_train, Target_train)


The ANOVA F-value, as I understand it, does not require a categorical response. (see scipy.stats.f_oneway) It is computing the value between the features. Why does f_classif require the response?

How does SelectKBest actually achieve a ranking of features based on the F-value when there should only be one F-value for a set of data?

Your question is really more about f_classif than SelectKBest. It's to drop duplicate labels; note the np.unique(y):

X, y = check_X_y(X, y, ['csr', 'csc', 'coo'])
args = [X[safe_mask(X, y == k)] for k in np.unique(y)]
return f_oneway(*args)


f_oneway still only gets passed the feature matrix, but a subset of it.

• Thanks for the link to the code, that's really interesting. I'm going to take a look and see if it makes sense. – Hobbes Jul 7 '16 at 20:30
• So the f_classif version of the f_oneway function and the scipy version of the f_oneway function are coded slightly differently and are not directly comparable. f_classif assumes more than one category and will treat features in each class as levels in a variable. Scipy assumes more than one level and treats each column (feature) as a level. So it seems to me that f_classif is not a real ANOVA. It also seems like ANOVA is discouraged for binary classification. Is it better to just use rank-sum tests to perform feature selection in binary (or really any amount) classification cases? – Hobbes Jul 8 '16 at 0:29
• I prefer $L_1$ or elastic net feature selection so I am unable to answer that question, unfortunately. – Emre Jul 8 '16 at 0:35
• I probably shouldn't say ANOVA discouraged for binary classification, but with just two classes it is the same as doing a t-test. Anyways, I've accepted your answer. – Hobbes Jul 8 '16 at 14:21