# How to grid search feature selection and neural network hyperparameters in the same grid?

I'm using the GridSearchCV () class from scikit to perform hyperparameter optimization in a sequential neural network. I've built a pipeline to also find the best number of features by putting a feature selector inside the pipeline. The problem is how to define the input_shape, since this depends on the k parameter from the feature selector. Is it possible to set the value of classifier__input_shape to be the same value (at all times) of feature_selector__feature__selector_k?

I've provided the correspondent piece of code below.

def create_model (learn_rate = 0.01, dropout_rate = 0.0, weight_constraint = 0, input_shape):
model = Sequential ()
model.add (Dense (units = 64, activation = 'relu',
input_shape = (input_shape, )))
model.add (Dense (32, activation = 'relu'))
model.add (Dense (1, activation = 'sigmoid'))
model.compile (loss = 'binary_crossentropy',
optimizer = Adam (lr = learn_rate),
metrics = ['accuracy'])#, metrics.CategoricalAccuracy ()])
return model

standard_scaler_features = remaining_features
my_scaler = StandardScaler ()
steps = list ()
steps.append (('scaler', my_scaler))
standard_scaler_transformer = Pipeline (steps)

my_feature_selector = SelectKBest ()
steps = list ()
steps.append (('feature_selector', my_feature_selector))
feature_selector_transformer = Pipeline (steps)

clf = KerasClassifier (build_fn = create_model, verbose = 2)
clf = Pipeline (steps = [('scaler', my_scaler),
('feature_selector', feature_selector_transformer),
('classifier', clf)],
verbose = True)

param_grid = {'feature_selector__feature_selector__score_func' : [f_classif],
'feature_selector__feature_selector__k' : [7, 9, 15],
'classifier__input_shape' : [7, 9, 15],
'classifier__epochs' : [2, 3, 4]}
cv = RepeatedStratifiedKFold (n_splits = 5, n_repeats = 1, random_state = STATE)
grid = GridSearchCV (estimator = clf, param_grid = param_grid, scoring = 'f1',
verbose = 1, n_jobs = 1, cv = cv)
grid_result = grid.fit (X_train_df, y_train_df)


And the error:

ValueError: Input 0 of layer sequential_9 is incompatible with the layer: expected axis -1 of input shape to have value 9 but received input with shape [None, 7]

I see two solutions:

1. either you pass a list of dictionnaries to param_grid avoiding irrelevant combinations
2. or you use a single variable in your pipeline for feature_selector__feature__selector_k and classifier__input_shape

First solution: you can generate the right list of combinations using something close to this:

param_grid = [
{
'feature_selector__feature_selector__score_func' : [f_classif],
'feature_selector__feature_selector__k' : [k],
'classifier__input_shape' : [k],
'classifier__dropout_rate' : [0.0, 0.5]
}
for k in [7, 9, 15]
]


Second solution, you can use a specific class that create your model when fitting based on the shape of X. Here is a code sample:

class MyKerasClf():
def predict(self, X):
y_pred_nn = self.clf.predict(X)
return np.array(y_pred_nn).flatten()

def create_model(self, learn_rate = 0.01, weight_constraint = 0 ):
model = Sequential ()
model.add (Dense (units = 64, activation = 'relu',
input_shape = (self.input_shape, )))
model.add (Dense (32, activation = 'relu'))
model.add (Dense (1, activation = 'sigmoid'))
model.compile (loss = 'binary_crossentropy',
optimizer = Adam (lr = learn_rate),
metrics = ['accuracy'])
return model

def fit(self, X, y, **kwargs):
self.input_shape = X.shape[1]
self.clf = KerasClassifier(build_fn = self.create_model, verbose = 2)
self.clf.fit(X, y, **kwargs)

def set_params(self, **params):
if 'dropout_rate' in params:
self.dropout_rate = params['dropout_rate']
else:
self.dropout_rate = 0.0


Then you can use the class in your pipeline

X, y = make_classification(n_features=50, n_redundant=0, n_informative=2,
random_state=42, n_clusters_per_class=1)

my_scaler = StandardScaler ()
steps = list ()
steps.append (('scaler', my_scaler))
standard_scaler_transformer = Pipeline (steps)

my_feature_selector = SelectKBest ()
steps = list ()
steps.append (('feature_selector', my_feature_selector))
feature_selector_transformer = Pipeline (steps)

# Create a specific clf
my_clf = MyKerasClf( )
pip_clf = Pipeline (steps = [('scaler', my_scaler),
('feature_selector', feature_selector_transformer),
('classifier', my_clf)],
verbose = True)

param_grid = {'feature_selector__feature_selector__score_func' : [f_classif],
'feature_selector__feature_selector__k' : [7, 15],
'classifier__dropout_rate' : [0.0, 0.5]
}
cv = RepeatedStratifiedKFold (n_splits = 5, n_repeats = 1, random_state = 42)
grid = GridSearchCV (estimator = pip_clf, param_grid = param_grid, scoring = 'f1',
verbose = 1, n_jobs = 1, cv = cv)
grid_result = grid.fit(X, y)


Note: I although added the dropout to be tested in the gridsearch as an example.

• Thanks, etiennedm. This certainly works. Ideally it should be possible to do this "in house", with a dependency option inside scikit. Sadly writing a custom classe would not solve the problem when another custom step in the pipeline gets added without modifying the class or writing another one. But I'll accept this answer if no one knows how to declare dependent parameters in the grid inside only scikit. Thanks again. – kaylani2 Jul 24 '20 at 23:23
• Your edit for the first solution was pretty much how I handled the problem at the time. Since it doesn't look like scikit has a feature to do this and the for loop, despite looking a bit hacky, solves the problem with little effort I'll accept the answer. Thanks again, etiennedm. – kaylani2 Jul 25 '20 at 22:21