1
$\begingroup$

I am trying to work on feature selection stage for my dataset.

I am a newbie to ML. I have around 60 columns and am trying to select top 15 features. I came to know about RFECV for which I wrote a code like as shown below. I am aware that n_features is present for RFE but it is missing for RFECV. Is there anyother way to assign the number of features to select?

model  = RandomForestClassifier(n_estimators=100, random_state=0)
# create the RFE model and select 15 attributes
rfe = RFECV(model,step=5, cv=5,min_features_to_select = 15,max_features_to_select = 15) # this doesn't work. `n_features=15` also doesn't work
rfe = rfe.fit(X_train_std, y_train)
# summarize the selection of the attributes
feat = rfe.support_
fret = rfe.ranking_
features = X.columns
print(features[feat].tolist())

Can someone help me to get top 15 features only? Where can I configure the n_features paramter?

Currently it displays more than 30 features. I don't really know how or from where does it get its number (30)?

$\endgroup$
1
$\begingroup$

That's the point of RFECV over RFE: the former selects the best number of features through cross-validation. If you want 15 features, use RFE instead (or some other feature selection method).

From the API docs,

cross-validated selection of the best number of features

and from the User Guide

RFECV performs RFE in a cross-validation loop to find the optimal number of features.


To respond to the comments (my response is a bit too long for comments):

Yes, RFECV is meant to produce the optimal number of features. RFE is run from the full feature set down to 1 feature on each of the cross-validation splits, then those models are scored on the test folds and averaged; then the best-scoring number of features can be taken and then RFE is run again down to that number of features. The thought is that (barring some special domain knowledge) RFECV is better than RFE, except that it may take rather longer to run. By definition, the RFECV's accuracy (on the CV-splitted training set) will be better than RFE with any other fixed number of features.

Now, the usual caveats apply. The number of features returned by RFECV may not necessarily generalize the best to unseen data, especially if that data isn't perfectly iid with the training data. Especially with small datasets, the splits made in the CV may affect the results, and perhaps going from the smaller CV train sets to the whole train set should allow more (or fewer?) features. And the top features on each of the splits may not be the same, so again when going to the full training set the question of increasing (decreasing?) number of features is reasonable.

$\endgroup$
  • $\begingroup$ When RFECV selects 30 features, does it mean that these 30 features play a role /capture the variation in target variable? I can use all these 30 features in a model? Upvoted for the help $\endgroup$ – The Great Dec 13 '19 at 15:30
  • 1
    $\begingroup$ I mean putting in all the 30 features will not reduce my model accuracy. Just trying to learn. Which one do you suggest? To just select top 15 or rely on RFECV optimum features? I know that data isn't shared. But just trying to understand from you as to what usually makes one to choose one over the other? $\endgroup$ – The Great Dec 13 '19 at 15:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.