I use Python and Weka to run feature selection on my dataset (91 predictor variables). I can see a huge difference (feature ranking) from different algorithms. And these results are still quite different from that derived from random forest or gradient boosting fitting. So how can I treat this gap or which algorithm I should trust? Is there any performance evaluation method or rule of thumb?
# Univariate Selection
import pandas
import numpy
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2
# feature extraction
test = SelectKBest(score_func=chi2, k=4)
fit = test.fit(X, y)
# summarize scores
numpy.set_printoptions(precision=3)
print(fit.scores_)
# Feature Extraction with RFE
from sklearn.feature_selection import RFE
# feature extraction
model = LogisticRegression()
rfe = RFE(model, 15)
fit = rfe.fit(X, y)
print("Num Features: %d" % fit.n_features_)
print("Selected Features: %s" % fit.support_)
print("Feature Ranking: %s" % fit.ranking_)
# VarianceThreshold
from sklearn.feature_selection import VarianceThreshold
sel = VarianceThreshold(threshold=(.8 * (1 - .8)))
sel.fit_transform(X)
idxs = sel.get_support(indices=True)
np.array(X)[:, idxs]