1
$\begingroup$

I am working on a feature selection for a binary classification problem with 977 records (and class proportion of 77:23). I already referred these two related posts - here and here. step size = 1 and number of columns is 61

I ran the below code for RFECV

from sklearn.feature_selection import RFECV
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold
estimator = RandomForestClassifier(random_state=1,class_weight='balanced',max_depth=5,max_features='sqrt',n_estimators=500)
selector = RFECV(estimator,step=1,cv=StratifiedKFold(10),scoring='f1',verbose=2)
selector = selector.fit(X_train,y_train)
plt.figure()
plt.xlabel("Number of features selected")
plt.ylabel("Cross validation score \n of number of selected features")
plt.plot(range(1, len(selector.grid_scores_) + 1), selector.grid_scores_)
plt.show()

And got the below output graph

enter image description here

As you can see that the best score is achieved when there are 10-15 features

But RFECV returns only 4 features as optimal number of features as shown below

print('Optimal number of features :', selector.n_features_)
print('Best features :', X_train.columns[selector.support_])

Optimal number of features : 4 Best features : Index(['F1','F2','F3','F4'],dtype='object')

but when I use Borutapy, Featurewiz, Sequential forward selection etc, my best feature subset is around 13-14.

So, am wondering whether I am interpreting the RFECV output incorrectly. Why is there some inconsistency here? Am I misinterpreting the output graph? Can you guide me on this?

update

enter image description here

enter image description here

enter image description here

Improve this question
$\endgroup$
4
  • $\begingroup$ Some of those fold curves are a bit higher around 5 than at 15; plot the average of the fold scores instead? $\endgroup$
    – Ben Reiniger ♦
    Mar 6, 2022 at 16:56
  • $\begingroup$ I also tried selector.cv_results_ - screenshot of which is updated. can help me interpret that please? I did see mean test score as 0.60 at 4th position. Does it mean it indicates 4 features? $\endgroup$
    – The Great
    Mar 7, 2022 at 2:54
  • $\begingroup$ I computed mean using np.average. Is this how it is done? - np.average(selector.grid_scores_, axis=1). You can find the screenshot above $\endgroup$
    – The Great
    Mar 7, 2022 at 3:01
  • $\begingroup$ Check the average grid score from the last image you have attached here. $\endgroup$
    – al imtiaz
    Dec 25, 2022 at 5:07

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.