2
$\begingroup$

Do the below codes do the same? If not, what are the differences?

fs = RFE(estimator=RandomForestClassifier(), n_features_to_select=10)
fs.fit(X, y)
print(fs.support_)
fs = SelectFromModel(RandomForestClassifier(), max_features=10)
fs.fit(X, y)
print(fs.support_)
fs= RandomForestClassifier(),
fs.fit(X, y)
print(fs.feature_importances_[:10,])
$\endgroup$

1 Answer 1

4
$\begingroup$

They are not the same.

As the name suggests, "recursive feature elimination" (RFE) recursively eliminates features, by fitting the model and throwing away the least-important one(s). After removing one feature, the next iteration may find the remaining features have changed order of importance. This is especially true in the presence of correlated features: they may split importance when included together, so might both be dropped by your second approach; but in RFE, one gets dropped at some point, but then the other one appears more important in the following iterations (since it no longer splits its importance with its now-dropped companion) and so is kept.

Your third approach doesn't do any feature selection; it just prints the first (not top) feature importances (according to the model fitted on all features).

$\endgroup$
3
  • $\begingroup$ Thanks for your answer, but what about using SelectFromModel(RandomForestClassifier(), max_features=10) it is equal to any of them? $\endgroup$
    – Niyaz
    Jun 4 at 21:53
  • 1
    $\begingroup$ Oh, for some reason that's what I thought your second code snippet was doing. Your second snippet actually doesn't do any intelligent selection, instead just printing importances of the first (not top) 10 features (from the model fitted on all). $\endgroup$
    – Ben Reiniger
    Jun 4 at 23:44
  • 1
    $\begingroup$ (I've edited the question and answer to add the SelectFromModel approach.) $\endgroup$
    – Ben Reiniger
    Jun 5 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.