I am working on a feature selection for a binary classification problem with 977 records (and class proportion of 77:23). I already referred these two related posts - here and here. step size = 1 and number of columns is 61

I ran the below code for RFECV

from sklearn.feature_selection import RFECV
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold
estimator = RandomForestClassifier(random_state=1,class_weight='balanced',max_depth=5,max_features='sqrt',n_estimators=500)
selector = RFECV(estimator,step=1,cv=StratifiedKFold(10),scoring='f1',verbose=2)
selector = selector.fit(X_train,y_train)
plt.xlabel("Number of features selected")
plt.ylabel("Cross validation score \n of number of selected features")
plt.plot(range(1, len(selector.grid_scores_) + 1), selector.grid_scores_)

And got the below output graph

enter image description here

As you can see that the best score is achieved when there are 10-15 features

But RFECV returns only 4 features as optimal number of features as shown below

print('Optimal number of features :', selector.n_features_)
print('Best features :', X_train.columns[selector.support_])

Optimal number of features : 4 Best features : Index(['F1','F2','F3','F4'],dtype='object')

but when I use Borutapy, Featurewiz, Sequential forward selection etc, my best feature subset is around 13-14.

So, am wondering whether I am interpreting the RFECV output incorrectly. Why is there some inconsistency here? Am I misinterpreting the output graph? Can you guide me on this?


enter image description here

enter image description here

enter image description here

  • $\begingroup$ Some of those fold curves are a bit higher around 5 than at 15; plot the average of the fold scores instead? $\endgroup$
    – Ben Reiniger
    Mar 6, 2022 at 16:56
  • $\begingroup$ I also tried selector.cv_results_ - screenshot of which is updated. can help me interpret that please? I did see mean test score as 0.60 at 4th position. Does it mean it indicates 4 features? $\endgroup$
    – The Great
    Mar 7, 2022 at 2:54
  • $\begingroup$ I computed mean using np.average. Is this how it is done? - np.average(selector.grid_scores_, axis=1). You can find the screenshot above $\endgroup$
    – The Great
    Mar 7, 2022 at 3:01
  • $\begingroup$ Check the average grid score from the last image you have attached here. $\endgroup$
    – al imtiaz
    Dec 25, 2022 at 5:07


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.