I am working on a feature selection for a binary classification problem with 977 records (and class proportion of 77:23). I already referred these two related posts - here and here. step size = 1 and number of columns is 61
I ran the below code for RFECV
from sklearn.feature_selection import RFECV
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold
estimator = RandomForestClassifier(random_state=1,class_weight='balanced',max_depth=5,max_features='sqrt',n_estimators=500)
selector = RFECV(estimator,step=1,cv=StratifiedKFold(10),scoring='f1',verbose=2)
selector = selector.fit(X_train,y_train)
plt.figure()
plt.xlabel("Number of features selected")
plt.ylabel("Cross validation score \n of number of selected features")
plt.plot(range(1, len(selector.grid_scores_) + 1), selector.grid_scores_)
plt.show()
And got the below output graph
As you can see that the best score is achieved when there are 10-15 features
But RFECV returns only 4 features as optimal number of features as shown below
print('Optimal number of features :', selector.n_features_)
print('Best features :', X_train.columns[selector.support_])
Optimal number of features : 4 Best features : Index(['F1','F2','F3','F4'],dtype='object')
but when I use Borutapy, Featurewiz, Sequential forward selection etc, my best feature subset is around 13-14.
So, am wondering whether I am interpreting the RFECV output incorrectly. Why is there some inconsistency here? Am I misinterpreting the output graph? Can you guide me on this?
update
selector.cv_results_
- screenshot of which is updated. can help me interpret that please? I did see mean test score as 0.60 at 4th position. Does it mean it indicates 4 features? $\endgroup$np.average
. Is this how it is done? -np.average(selector.grid_scores_, axis=1)
. You can find the screenshot above $\endgroup$