I use Python and Weka to run feature selection on my dataset (91 predictor variables). I can see a huge difference (feature ranking) from different algorithms. And these results are still quite different from that derived from random forest or gradient boosting fitting. So how can I treat this gap or which algorithm I should trust? Is there any performance evaluation method or rule of thumb?

enter image description here

# Univariate Selection
import pandas
import numpy
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2

# feature extraction
test = SelectKBest(score_func=chi2, k=4)
fit = test.fit(X, y)
# summarize scores

# Feature Extraction with RFE
from sklearn.feature_selection import RFE

# feature extraction
model = LogisticRegression()
rfe = RFE(model, 15)
fit = rfe.fit(X, y)
print("Num Features: %d" % fit.n_features_)
print("Selected Features: %s" % fit.support_)
print("Feature Ranking: %s" % fit.ranking_)

# VarianceThreshold
from sklearn.feature_selection import VarianceThreshold
sel = VarianceThreshold(threshold=(.8 * (1 - .8)))

idxs = sel.get_support(indices=True)
np.array(X)[:, idxs]
  • $\begingroup$ Cross validate! $\endgroup$
    – Emre
    Jan 10 '17 at 21:10

It is generally not best practice to trust ML techniques to do the work for you. A data scientist knowns on what principles the methods that he or she applies work, and then stills tries multiple in the same setting, to see which methods best apply to the domain at hand. How well certain measures work can be a clue to the structure of your data.

Right off the bat it is important to understand that Weka offers 3 kinds of feature selection methods: Filters, a Wrapper and PCA. Filters apply criteria (mostly based on a measure of how it divides a set) for feature suitedness (variations on how well they divide the search space) , wrappers use a learning algorithm for their evaluation and PCA doesn't so much select features as that it recombines features to maximize variance in the dimensions.

That being said: I suggest you start looking at InformationGainEval a method that uses information gain to select attributes. It is quite popular and easy to understand. You can find a good explantation here: https://stackoverflow.com/questions/33982943/how-the-selection-happens-in-infogainattributeeval-in-weka-feature-selection. Than work your way through the others (the documentation is accessible from the GUI).

  • $\begingroup$ Agree - InfoGain is most closely related to what decision trees do to pick which features to use for split nodes. But in general greedy feature selection is not guaranteed to give you an optimal set of features. If you are planning on using a linear model like logistic regression you might consider using LASSO instead - it'll pick the best set of features in combination rather than one at a time. $\endgroup$
    – tom
    Nov 17 '17 at 6:46

I would look at Sebastian Raschka's discussion on different types of feature selection methods.. In short, there appear to be three categories (each with advantages and disadvantages):

  1. Filters
  2. Wrappers
  3. Embedded Methods

Sebastian goes on to discuss specific feature selection techniques (i.e PCA) and describes the process in 3 simple steps - might be worth looking into

NOTE: I would typically make this a 'comment', but an unable to do so due to my low reputation within this forum


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.