# Which feature selection I should trust?

I use Python and Weka to run feature selection on my dataset (91 predictor variables). I can see a huge difference (feature ranking) from different algorithms. And these results are still quite different from that derived from random forest or gradient boosting fitting. So how can I treat this gap or which algorithm I should trust? Is there any performance evaluation method or rule of thumb?

# Univariate Selection
import pandas
import numpy
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2

# feature extraction
test = SelectKBest(score_func=chi2, k=4)
fit = test.fit(X, y)
# summarize scores
numpy.set_printoptions(precision=3)
print(fit.scores_)

# Feature Extraction with RFE
from sklearn.feature_selection import RFE

# feature extraction
model = LogisticRegression()
rfe = RFE(model, 15)
fit = rfe.fit(X, y)
print("Num Features: %d" % fit.n_features_)
print("Selected Features: %s" % fit.support_)
print("Feature Ranking: %s" % fit.ranking_)

# VarianceThreshold
from sklearn.feature_selection import VarianceThreshold
sel = VarianceThreshold(threshold=(.8 * (1 - .8)))
sel.fit_transform(X)

idxs = sel.get_support(indices=True)
np.array(X)[:, idxs]

• Cross validate!
– Emre
Commented Jan 10, 2017 at 21:10

It is generally not best practice to trust ML techniques to do the work for you. A data scientist knowns on what principles the methods that he or she applies work, and then stills tries multiple in the same setting, to see which methods best apply to the domain at hand. How well certain measures work can be a clue to the structure of your data.

Right off the bat it is important to understand that Weka offers 3 kinds of feature selection methods: Filters, a Wrapper and PCA. Filters apply criteria (mostly based on a measure of how it divides a set) for feature suitedness (variations on how well they divide the search space) , wrappers use a learning algorithm for their evaluation and PCA doesn't so much select features as that it recombines features to maximize variance in the dimensions.

That being said: I suggest you start looking at InformationGainEval a method that uses information gain to select attributes. It is quite popular and easy to understand. You can find a good explantation here: https://stackoverflow.com/questions/33982943/how-the-selection-happens-in-infogainattributeeval-in-weka-feature-selection. Than work your way through the others (the documentation is accessible from the GUI).

• Agree - InfoGain is most closely related to what decision trees do to pick which features to use for split nodes. But in general greedy feature selection is not guaranteed to give you an optimal set of features. If you are planning on using a linear model like logistic regression you might consider using LASSO instead - it'll pick the best set of features in combination rather than one at a time.
– tom
Commented Nov 17, 2017 at 6:46

I would look at Sebastian Raschka's discussion on different types of feature selection methods.. In short, there appear to be three categories (each with advantages and disadvantages):

1. Filters
2. Wrappers
3. Embedded Methods

Sebastian goes on to discuss specific feature selection techniques (i.e PCA) and describes the process in 3 simple steps - might be worth looking into

NOTE: I would typically make this a 'comment', but an unable to do so due to my low reputation within this forum