Mismatch between optimum features and grid scores using RFECV?

Question

I have a dataset with 5K columns focused on binary classification. I have more than 60 columns. I am trying to find the best features through RFECV approach. Though it produces 30 optimum features, when I plot in graph, I only see 12 features. Please see my code and plot below

model  = RandomForestClassifier(n_estimators=100, random_state=0)
model_b = LinearSVC(class_weight='balanced',max_iter=1000)
# create the RFE model and select 15 attributes
rfe = RFECV(model,step=5,cv=5)
rfe = rfe.fit(X_train_std, y_train)
# summarize the selection of the attributes
feat = rfe.support_
fret = rfe.ranking_
features = X.columns
print(rfe.n_features_) # this returns 30 as output

print(rfe.grid_scores_) this produces the below output

I was expecting to see the grid scores for 30 features and in the plot, I was expecting to see the x-axis to have 30 features as well. But it shows only for 12 features. Similarly if I had only 19 features in my dataset, RFECV returns all 19 as optimal feature which is fine. But again in grid score it only shows 4

q1) Does this mean beyond 12 features, there is no increase in model accuracy?

q2) I assume grid_scores are nothing but weightage/rank which indicates the influence that a feature has on outcome. But how do I get the name of this 12 features?

q3) Wwhy does it show optimum no of features as 30 but grid scores is shown only for 12.

Can you help me with these question please?

Answer 1

RFE stands for Recursive Feature Elimination, meaning that the search starts off with a full set of features and with each step some features (in your case, 5) are dropped in an attempt to improve the predictive power of the model. If we take a closer look at sklearn's RFECV reference, it states:

grid_scores_ : array of shape [n_subsets_of_features]

The cross-validation scores such that grid_scores_[i] corresponds to the CV score of the i-th subset of features.

The i-th subset of features being referred to is the group of features being evaluated at that particular step of the search. Judging by your output (and the plot), only 12 steps were needed to figure out which 30 features are optimal and that no further improvements could be found using the chosen approach - that's why you see an increase in CV score followed by a plateau. Your assumption in q1 is almost correct, but you are looking at algorithm steps, not features. This is simply an issue of misinterpretation, so is q3.

As for q2, you need to look at support_ attribute of the RFECV object. It returns a boolean mask that corresponds to X.columns (if True, then that feature is selected, if False, then not). You can also look at ranking_, but it provides information on the "goodness" of discarded features only (all selected features are ranked 1). The importances of selected features are model-specific. For RandomForestClassifier, you can look at feature_importances_. For LinearSVC, look at coef_.

Hopefully this answers your question. Good luck with the selection!