I am working on a feature selection for a binary classification problem with 977 records (and class proportion of 77:23). I already referred these two related posts - here and here. step size = 1 and number of columns is 61
I ran the below code for
from sklearn.feature_selection import RFECV from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import StratifiedKFold estimator = RandomForestClassifier(random_state=1,class_weight='balanced',max_depth=5,max_features='sqrt',n_estimators=500) selector = RFECV(estimator,step=1,cv=StratifiedKFold(10),scoring='f1',verbose=2) selector = selector.fit(X_train,y_train) plt.figure() plt.xlabel("Number of features selected") plt.ylabel("Cross validation score \n of number of selected features") plt.plot(range(1, len(selector.grid_scores_) + 1), selector.grid_scores_) plt.show()
And got the below output graph
As you can see that the best score is achieved when there are 10-15 features
But RFECV returns only 4 features as optimal number of features as shown below
print('Optimal number of features :', selector.n_features_) print('Best features :', X_train.columns[selector.support_])
Optimal number of features : 4 Best features : Index(['F1','F2','F3','F4'],dtype='object')
but when I use Borutapy, Featurewiz, Sequential forward selection etc, my best feature subset is around 13-14.
So, am wondering whether I am interpreting the RFECV output incorrectly. Why is there some inconsistency here? Am I misinterpreting the output graph? Can you guide me on this?