4
$\begingroup$

SelectKBest(f_classif, k), where k is the number of features to select, is often used for feature selection, however, I am having trouble finding descriptive documentation on how it works. A sample of how this works is below:

model = SelectKBest(f_classif, k)
model.fit_transform(X_train, Target_train)

The ANOVA F-value, as I understand it, does not require a categorical response. (see scipy.stats.f_oneway) It is computing the value between the features. Why does f_classif require the response?

How does SelectKBest actually achieve a ranking of features based on the F-value when there should only be one F-value for a set of data?

$\endgroup$
2
$\begingroup$

Your question is really more about f_classif than SelectKBest. It's to drop duplicate labels; note the np.unique(y):

X, y = check_X_y(X, y, ['csr', 'csc', 'coo'])
args = [X[safe_mask(X, y == k)] for k in np.unique(y)]
return f_oneway(*args)

f_oneway still only gets passed the feature matrix, but a subset of it.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thanks for the link to the code, that's really interesting. I'm going to take a look and see if it makes sense. $\endgroup$ – Hobbes Jul 7 '16 at 20:30
  • $\begingroup$ So the f_classif version of the f_oneway function and the scipy version of the f_oneway function are coded slightly differently and are not directly comparable. f_classif assumes more than one category and will treat features in each class as levels in a variable. Scipy assumes more than one level and treats each column (feature) as a level. So it seems to me that f_classif is not a real ANOVA. It also seems like ANOVA is discouraged for binary classification. Is it better to just use rank-sum tests to perform feature selection in binary (or really any amount) classification cases? $\endgroup$ – Hobbes Jul 8 '16 at 0:29
  • $\begingroup$ I prefer $L_1$ or elastic net feature selection so I am unable to answer that question, unfortunately. $\endgroup$ – Emre Jul 8 '16 at 0:35
  • $\begingroup$ I probably shouldn't say ANOVA discouraged for binary classification, but with just two classes it is the same as doing a t-test. Anyways, I've accepted your answer. $\endgroup$ – Hobbes Jul 8 '16 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.