6
$\begingroup$

SelectKBest(f_classif, k), where k is the number of features to select, is often used for feature selection, however, I am having trouble finding descriptive documentation on how it works. A sample of how this works is below:

model = SelectKBest(f_classif, k)
model.fit_transform(X_train, Target_train)

The ANOVA F-value, as I understand it, does not require a categorical response. (see scipy.stats.f_oneway) It is computing the value between the features. Why does f_classif require the response?

How does SelectKBest actually achieve a ranking of features based on the F-value when there should only be one F-value for a set of data?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Your question is really more about f_classif than SelectKBest. It's to drop duplicate labels; note the np.unique(y):

X, y = check_X_y(X, y, ['csr', 'csc', 'coo'])
args = [X[safe_mask(X, y == k)] for k in np.unique(y)]
return f_oneway(*args)

f_oneway still only gets passed the feature matrix, but a subset of it.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the link to the code, that's really interesting. I'm going to take a look and see if it makes sense. $\endgroup$
    – Hobbes
    Commented Jul 7, 2016 at 20:30
  • $\begingroup$ So the f_classif version of the f_oneway function and the scipy version of the f_oneway function are coded slightly differently and are not directly comparable. f_classif assumes more than one category and will treat features in each class as levels in a variable. Scipy assumes more than one level and treats each column (feature) as a level. So it seems to me that f_classif is not a real ANOVA. It also seems like ANOVA is discouraged for binary classification. Is it better to just use rank-sum tests to perform feature selection in binary (or really any amount) classification cases? $\endgroup$
    – Hobbes
    Commented Jul 8, 2016 at 0:29
  • $\begingroup$ I prefer $L_1$ or elastic net feature selection so I am unable to answer that question, unfortunately. $\endgroup$
    – Emre
    Commented Jul 8, 2016 at 0:35
  • $\begingroup$ I probably shouldn't say ANOVA discouraged for binary classification, but with just two classes it is the same as doing a t-test. Anyways, I've accepted your answer. $\endgroup$
    – Hobbes
    Commented Jul 8, 2016 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.