How to implement Python's MLPClassifier with gridsearchCV?

Question

I am trying to implement Python's MLPClassifier with 10 fold cross-validation using gridsearchCV function. Here is a chunk of my code:

parameters={
'learning_rate': ["constant", "invscaling", "adaptive"],
'hidden_layer_sizes': [(100,1), (100,2), (100,3)],
'alpha': [10.0 ** -np.arange(1, 7)],
'activation': ["logistic", "relu", "Tanh"]
}

clf
= gridSearchCV(estimator=MLPClassifier,param_grid=parameters,n_jobs=-1,verbose=2,cv=10)

Though,I am not sure if hidden_layer_sizes: [(100,1), (100,2), (100,3)] is correct. Here, I am trying to tune 'hidden layer size' & 'number of neurons'. I would like to give this 'tuple' parameter for hidden_layer_sizez: 1, 2, 3, and neurons: 10, 20, 30,...,100.

But I do not know if it is the correct way to do it. Therefore, I am choosing default neurons to be 100 in each layer.

Can anyone advise please?

Answer 1

A tuple of the form $(i_1, i_2, i_3, ... , i_n)$ gives you a network with $n$ hidden layers, where $i_k$ gives you the number of neurons in the $k$th hidden layer.

If you want three hidden layers with $10,30$ and $20$ neurons, your tuple would need to look like $(10,30,20)$.

$(100,1)$ would mean that the second hidden layer only has one neuron.

Answer 2

You can implement MLPClassifier with GridSearchCV in scikit-learn as follows (other parameters are also available):

GRID = [
    {'scaler': [StandardScaler()],
     'estimator': [MLPClassifier(random_state=RANDOM_SEED)],
     'estimator__solver': ['adam'],
     'estimator__learning_rate_init': [0.0001],
     'estimator__max_iter': [300],
     'estimator__hidden_layer_sizes': [(500, 400, 300, 200, 100), (400, 400, 400, 400, 400), (300, 300, 300, 300, 300), (200, 200, 200, 200, 200)],
     'estimator__activation': ['logistic', 'tanh', 'relu'],
     'estimator__alpha': [0.0001, 0.001, 0.005],
     'estimator__early_stopping': [True, False]
     }
]

PIPELINE = Pipeline([('scaler', None), ('estimator', MLPClassifier())])

You can then run GridSearch as the following:

grid_search = GridSearchCV(estimator=PIPELINE, param_grid=GRID, 
                            scoring=make_scorer(accuracy_score),# average='macro'), 
                            n_jobs=-1, cv=split, refit=True, verbose=1, 
                            return_train_score=False)

grid_search.fit(X, y)