I use Python and Weka to run feature selection on my dataset (91 predictor variables). I can see a huge difference (feature ranking) from different algorithms. And these results are still quite different from that derived from random forest or gradient boosting fitting. So how can I treat this gap or which algorithm I should trust? Is there any performance evaluation method or rule of thumb?
# Univariate Selection import pandas import numpy from sklearn.feature_selection import SelectKBest from sklearn.feature_selection import chi2 # feature extraction test = SelectKBest(score_func=chi2, k=4) fit = test.fit(X, y) # summarize scores numpy.set_printoptions(precision=3) print(fit.scores_) # Feature Extraction with RFE from sklearn.feature_selection import RFE # feature extraction model = LogisticRegression() rfe = RFE(model, 15) fit = rfe.fit(X, y) print("Num Features: %d" % fit.n_features_) print("Selected Features: %s" % fit.support_) print("Feature Ranking: %s" % fit.ranking_) # VarianceThreshold from sklearn.feature_selection import VarianceThreshold sel = VarianceThreshold(threshold=(.8 * (1 - .8))) sel.fit_transform(X) idxs = sel.get_support(indices=True) np.array(X)[:, idxs]