Sklearn feature selection stopping criterion (SelectFromModel)

Question

Sklearn has several functions for feature selection that lets the user determine the size of the chosen subset. An example of this is SelectKBest where the user determines the value of "k", which is the number of top performing features.

Does anyone know what stopping criterion SelectFromModel uses when it selects a feature subset? The documentation mentiones a "threshold"-parameter that determines which features are important enough, and that this parameter is set to "median" OR "mean" by default.

Answer 1

Some regression/classification models can also calculate feature importances - for example RandomForestClassifier models have a property feature_importances_ and LogisticRegression models have a coef_ property. There are many more models that can provide feature importances - but all of them either have a coef_ or feature_importances_ property.

What SelectFromModel does is to check if the object you pass as "estimator" has one of these two properties and to return features where this property is higher than the specified threshold (or the mean/median).

For example if you pass a RandomForestClassifier to SelectFromModel, it will return all features where the random forest's feature_importances_ property is higher than the specified threshold. The same happens if you pass a LogisticRegression model, except that it'll compare the coef_ property with the threshold instead.

Selecting the best value for the threshold can be done using a grid- or randomized search.