Feature selection filter methods

Question

I am confused about when to use which filter methods for feature selection. I tried to learn them through online resources and found methods like chi-square, variance threshold, F-test, Mutual information, etc. As I read through the methods, I got confused with which one I should use. How do I know if the method really selected the most relevant features and did not drop any features that were, in fact, contributing to the cause of the output?

Answer 1

In general, you need to wrap the feature selection filter method within a cross validation scheme. Here is an example of such an algorithm, assuming you are also tuning hyperparameters.

1. Split the data into training and testing.

2. Split the training set only into k folds.

3. For each iteration of hyperparameter tuning, use cross validation, i.e. do:

For each of the k folds, for iterations i = 1 to k, do:

1) Let the validation set be fold i, and the "inner training" set be all folds not equal to fold i.

2) Using your filtering method, select some subset of your features using some cutoff p on the inner training set. For example, I might filter out variables that have absolute correlation with the target variable y less than p = 0.3.

3) Fit a model.

4) Predict the validation set = fold i. Score the model, i.e. evaluate your function you are trying to optimizer like log loss, mean squared/absolute error, Brier, etc. Save the score.

end

After you are done with your cross validation for one set of hyperparameters, take the mean score over all of the k folds. If the score is better than a previous set of hyperparameters found in a prior iteration, save these hyperparameters.

Note: you can choose to have p, your filter cutoff, also be a hyperparameter here. For simplicity I kept it constant but in general you will probably do better if you tune the filtering cutoff as well.

end

4. Using the same (or best, if you let it be a hyperparameter and optimized over it in step 3) filter cutoff p, and the best hyperparameters that you found in the cross validation process in step 3, use the training set created in step 1, and fit your "final" model. That is, filter your training set with the cutoff value p using the exact same filtering method you did in step 3. Then, fit a model to this data set using the optimal hyper parameters, also found in step 3.

5. Predict the test set using your fitted model in step 4. Score the model.

Finally, repeat the entire process again but this time, don't do any filter selection method (omit the filtering process). Use the exact same seeds in your cross validation and at the initial split in step 1 to make sure you are splitting your data the same way from when you used filtering. Now, compare test set scores and see for yourself if there is any benefit to using your filtering method.

In general, I am not a fan of filter based methods because they all introduce some arbitrary cutoff = p and also are all univariate (they don't take into consideration other variables in the model, just in isolation). That being said, they are fast and they might work for your problem.

EDIT: Also, depending on how much data you have and how variable your validation scores are, multiple repeats of the entire algorithm above (steps 1 through 5) might be necessary in order to obtain valid comparisons between using filtering and not using filtering. You would repeat the algorithm above say, N times, once with filtering and once without for a total of 2N times. You would then calculate an average of averages (find the average score over N repeats using filtering, and find the average score over the same N repeats with no filtering). Then, you could compare scores, form confidence intervals, etc. for a more trustworthy (but also more time consuming) comparison.

Hope this helps.

Answer 2

Check Recursive Feature Elimination (RFE), if you have a labeled data.

It is exacly point out your question : How do I know if the method really selected the most relevant features and did not drop any features that were, in fact, contributing to the cause of the output?

https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html

From docs:

Given an external estimator that assigns weights to features (e.g., the coefficients of a linear model), the goal of recursive feature elimination (RFE) is to select features by recursively considering smaller and smaller sets of features. First, the estimator is trained on the initial set of features and the importance of each feature is obtained either through a coef_ attribute or through a feature_importances_ attribute. Then, the least important features are pruned from current set of features. That procedure is recursively repeated on the pruned set until the desired number of features to select is eventually reached.

Hope it helps!