Does sklearn perform feature selection within cross validation?

Question

I would like to add a feature selector on my pipeline and use gridsearchcv to tune both the hyperparameters of the selector and the classifier(s).

I am wondering if sklearn performs feature selection within cross validation. For example lets say that I want to perform forward selection using the SequentialFeatureSelector and one of the configurations of the grid is a random forest with 150 estimators and min_samples_leaf 10.

Does the SequentialFeatureSelector or any other Selector in sklearn performs the feature selection on each fold?

That is, if i have 3 features and perform 5-fold cv, does this mean that sklearn will train 15 models in order to select the first feature?

Edit I am adding the code that I have currently written. This performs a univariate feature selection with SelectKBest.

import numpy as np
import joblib
import json
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, SequentialFeatureSelector
from sklearn.feature_selection import f_classif, mutual_info_classif
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV, train_test_split

# For reproducible results
np.random.seed(1)

# Importing the df
df = pd.read_csv('data.csv')
df.set_index('MOF Name', inplace=True)

X, y = df.iloc[:, :-1], df.iloc[:, -1] # Target is the last column

# Split to train and test set with stratification
X_train, X_test, y_train, y_test = train_test_split(
    X, y, stratify=y, train_size=0.8
    )

# Define the pipeline
steps = [
    ('selector', SelectKBest()),
    ('scaler', StandardScaler()),
    ('classifier', SVC())
    ]

pipeline = Pipeline(steps)

# Selector(s) hyperparameters
kbest_params = {
    'selector__k': [2, 3, 4],
    'selector__score_func': [f_classif, mutual_info_classif],
}

sequential_params = ...

# SVC hyperparameters
svc_params = {
    'classifier__kernel': ['linear', 'rbf'],
    'classifier__C': [0.001, 0.01, 0.1, 1.0, 10, 100],
    'classifier__gamma': [0.01, 1, 5, 10],
}

# RF hyperparameters
rf_params = {
    'classifier__n_estimators': [100, 150, 200],
    'classifier__min_samples_leaf': [1, 5, 10, 20],
    'classifier__class_weight': ['balanced', None],
    'classifier__max_features': ['sqrt', None],
}

# Define the grid of hyperparameters
param_grid = [
    {**kbest_params, **clf_params} for clf_params in [svc_params, rf_params]
    ]

# Perform cross-valdiation for tuning hyperparameters
grid_search = GridSearchCV(
estimator=pipeline, param_grid=param_grid, scoring='roc_auc', verbose=2
    )

grid_search.fit(X_train, y_train)

# Save df and final model
df = [X_train, y_train, X_test, y_test, grid_search]
with open('cv_results', 'wb') as fhand:
    joblib.dump(df, fhand)

I have set verbose=2 and I get the output

Fitting 5 folds for each of 288 candidates, totalling 1440 fits

which is consistent with the number of hyperparameters that I am examining (note that parameters for random forest have been commented out).

Is there any idea on how to perform feature selection in a similar way? That is, perform SequentialFeatureSelector for each configuration of the grid within cross validation?

Answer 1

Thanks for providing code. Are you sure you really want to: perform SequentialFeatureSelector for each configuration of the grid within cross validation?

Do you realize that will be super slow because SequentialFeatureSelector will train many models (it is iterative method with number of iterations == number of features you have and each iteration will do yet another cross-validation which trains multiple models)?

Normally what people do in your situation is: First do the hyperparameter tuning to find good hyperparameters for full dataset, where in your case you should just treat the SelectKBest arguments as additional hyperparameters to optimize (consider different values of k and different values of score_func: f_classif, mutual_info_classif, chi2).

Then fix your estimator with the best hyperparameter values and run SequentialFeatureSelector with these hyperparameters fixed. Finally after you have finalized your feature-set, hold it fixed, and you can run the hyperparameter optimization one more time with estimator fit to reduced feature set (this time removing SelectKBest from your pipeline).