How to compare the performance of feature selection methods?

Question

There are several feature selection / variable selection approaches (see for example Guyon & Elisseeff, 2003; Liu et al., 2010):

• filter methods (e.g., correlation-based, entropy-based, random forest importance based),
• wrapper methods (e.g., forward-search, hill-climbing search), and
• embedded methods where the feature selection is part of the model learning.

Many published algorithms are also implemented in the machine learning tools like R, Python, etc.

What would be an appropriate method to compare different feature selection algorithms and to select the best method for a given problem / dataset? A further question would be, whether there are any metrics known that measure the performance of feature selection algorithms?

Answer 1

This is a hard problem and researchers are making a lot of progress.

If you're looking for supervised feature selection, I'd recommend LASSO and its variants. Evaluation of the algorithm is very straightforward with supervised learning: the performance of whichever metric you choose on test data.

Two major caveats of LASSO are that (1) the selected features will not automatically detect an interaction, so you have to craft all of your features a priori (i.e., before running them through the model) and (2) LASSO will not identify non-linear relationships (e.g., a quadratic relationship).

A way to try and get past these two caveats is to use Gradient Boosted Machines which does feature selection automatically. It's worth noting the statistical properties of GBM are a little more ambiguous than that of the LASSO.

If you're looking for unsupervised feature selection, it seems there's a similar regularization approach used by these researchers, but evaluation in this particular case becomes less obvious. People try a lot of different things like PCA/SVD or K-Means which ultimately will try to find a linear approximation to the data.

In that case, the typical measures of performance are the reconstruction error or the RMSE of the clusters.

In terms of software, R and Python both have GBM, LASSO, K-Means, SVD, and PCA. GLMNET and XGBoost in R and Sklearn for Python are the relevant libraries.

Answer 2

I always consider features selection as a step to a final result.

Hereunder, I somehow mix features selection and dimensionality reduction, which might have some goals and can be confused.

Some typical uses:

• reduction of computations in machine learning: the quality of the selection is a factor of the final learning result and also, obviously, the speed to get that learning done

• visualization/understanding of the data, where you combine eventually multiple dimensions. It is good when it doesn't hides interesting stuffs, and when that's understandable

• simplification of the learning results, still to make them understandable (eg root cause analysis). Good if simple but still sufficient in terms of quality

• controlling over fitting, as the previous reply suggests

• ...

So, I don't think there's general rule (as always in ML), but this is a case by case problem.

Just a personal belief...

Answer 3

It's very dependent on the specific situation and the problem you want to solve. There exist some general rules, for example wrapper methods are more flexible and also more prone to overfitting.

Feature selection performance can be evaluated by the overall performance of learning task for example one can select features with different methods and then use these different feature sets for classification and compare the precision of obtained classifiers.

Another important factor in some scenarios like some biological applications is the interpretability of selected features and the results, for example in a clustering problem, meaning of selected features and resulted clusters is a very important measure of performance.

Answer 4

You would have to run a set of artificial tests, trying to detect relevant features using different methods while knowing in advance which subsets of input variables affect the output variable.

Good trick would be to keep a set of random input variables with different distributions and make sure your feature selection algos indeed tag them as not relevant.

Another trick would be to make sure that after permuting rows the variables tagged as relevant stop being classified as relevant.

Above said applies to both filter and wrapper approaches.

Also be sure to handle cases when if taken separately (one by one) variables do not show any influence on the target, but when taken jointly reveal a strong dependence. Example would be a well-known XOR problem (check out the Python code):

import numpy as np
import matplotlib.pyplot as plt
from sklearn.feature_selection import f_regression, mutual_info_regression,mutual_info_classif

x=np.random.randn(5000,3)
y=np.where(np.logical_xor(x[:,0]>0,x[:,1]>0),1,0)

plt.scatter(x[y==1,0],x[y==1,1],c='r',marker='x')
plt.scatter(x[y==0,0],x[y==0,1],c='b',marker='o')
plt.show()
print(mutual_info_classif(x, y)) 

Output:

[ 0. 0. 0.00429746]

So, presumably powerful (but univariate) filtering method (computing of mutual information between out- and input variables) was not able to detect any relationships in the dataset. Whereas we know for sure it's a 100% dependency and we can predict Y with 100% accuracy knowing X.

Good idea would be to create a kind of benchmark for features selection methods, does anyone want to participate?