I would like to add a feature selector on my pipeline and use gridsearchcv to tune both the hyperparameters of the selector and the classifier(s).
I am wondering if sklearn performs feature selection within cross validation. For example lets say that I want to perform forward selection using the SequentialFeatureSelector
and one of the configurations of the grid is a random forest with 150 estimators and min_samples_leaf 10.
Does the SequentialFeatureSelector
or any other Selector
in sklearn performs the feature selection on each fold?
That is, if i have 3 features and perform 5-fold cv, does this mean that sklearn will train 15 models in order to select the first feature?
Edit
I am adding the code that I have currently written. This performs a univariate feature selection with SelectKBest
.
import numpy as np
import joblib
import json
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, SequentialFeatureSelector
from sklearn.feature_selection import f_classif, mutual_info_classif
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV, train_test_split
# For reproducible results
np.random.seed(1)
# Importing the df
df = pd.read_csv('data.csv')
df.set_index('MOF Name', inplace=True)
X, y = df.iloc[:, :-1], df.iloc[:, -1] # Target is the last column
# Split to train and test set with stratification
X_train, X_test, y_train, y_test = train_test_split(
X, y, stratify=y, train_size=0.8
)
# Define the pipeline
steps = [
('selector', SelectKBest()),
('scaler', StandardScaler()),
('classifier', SVC())
]
pipeline = Pipeline(steps)
# Selector(s) hyperparameters
kbest_params = {
'selector__k': [2, 3, 4],
'selector__score_func': [f_classif, mutual_info_classif],
}
sequential_params = ...
# SVC hyperparameters
svc_params = {
'classifier__kernel': ['linear', 'rbf'],
'classifier__C': [0.001, 0.01, 0.1, 1.0, 10, 100],
'classifier__gamma': [0.01, 1, 5, 10],
}
# RF hyperparameters
rf_params = {
'classifier__n_estimators': [100, 150, 200],
'classifier__min_samples_leaf': [1, 5, 10, 20],
'classifier__class_weight': ['balanced', None],
'classifier__max_features': ['sqrt', None],
}
# Define the grid of hyperparameters
param_grid = [
{**kbest_params, **clf_params} for clf_params in [svc_params, rf_params]
]
# Perform cross-valdiation for tuning hyperparameters
grid_search = GridSearchCV(
estimator=pipeline, param_grid=param_grid, scoring='roc_auc', verbose=2
)
grid_search.fit(X_train, y_train)
# Save df and final model
df = [X_train, y_train, X_test, y_test, grid_search]
with open('cv_results', 'wb') as fhand:
joblib.dump(df, fhand)
I have set verbose=2
and I get the output
Fitting 5 folds for each of 288 candidates, totalling 1440 fits
which is consistent with the number of hyperparameters that I am examining (note that parameters for random forest have been commented out).
Is there any idea on how to perform feature selection in a similar way? That is, perform SequentialFeatureSelector
for each configuration of the grid within cross validation?