Which feature selection I should trust?

Question

I use Python and Weka to run feature selection on my dataset (91 predictor variables). I can see a huge difference (feature ranking) from different algorithms. And these results are still quite different from that derived from random forest or gradient boosting fitting. So how can I treat this gap or which algorithm I should trust? Is there any performance evaluation method or rule of thumb?

# Univariate Selection
import pandas
import numpy
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2

# feature extraction
test = SelectKBest(score_func=chi2, k=4)
fit = test.fit(X, y)
# summarize scores
numpy.set_printoptions(precision=3)
print(fit.scores_)


# Feature Extraction with RFE
from sklearn.feature_selection import RFE

# feature extraction
model = LogisticRegression()
rfe = RFE(model, 15)
fit = rfe.fit(X, y)
print("Num Features: %d" % fit.n_features_)
print("Selected Features: %s" % fit.support_)
print("Feature Ranking: %s" % fit.ranking_)



# VarianceThreshold
from sklearn.feature_selection import VarianceThreshold
sel = VarianceThreshold(threshold=(.8 * (1 - .8)))
sel.fit_transform(X)

idxs = sel.get_support(indices=True)
np.array(X)[:, idxs]

Answer 1

It is generally not best practice to trust ML techniques to do the work for you. A data scientist knowns on what principles the methods that he or she applies work, and then stills tries multiple in the same setting, to see which methods best apply to the domain at hand. How well certain measures work can be a clue to the structure of your data.

Right off the bat it is important to understand that Weka offers 3 kinds of feature selection methods: Filters, a Wrapper and PCA. Filters apply criteria (mostly based on a measure of how it divides a set) for feature suitedness (variations on how well they divide the search space) , wrappers use a learning algorithm for their evaluation and PCA doesn't so much select features as that it recombines features to maximize variance in the dimensions.

That being said: I suggest you start looking at InformationGainEval a method that uses information gain to select attributes. It is quite popular and easy to understand. You can find a good explantation here: https://stackoverflow.com/questions/33982943/how-the-selection-happens-in-infogainattributeeval-in-weka-feature-selection. Than work your way through the others (the documentation is accessible from the GUI).

Answer 2

I would look at Sebastian Raschka's discussion on different types of feature selection methods.. In short, there appear to be three categories (each with advantages and disadvantages):

1. Filters
2. Wrappers
3. Embedded Methods

Sebastian goes on to discuss specific feature selection techniques (i.e PCA) and describes the process in 3 simple steps - might be worth looking into

NOTE: I would typically make this a 'comment', but an unable to do so due to my low reputation within this forum