6
$\begingroup$

I am trying to implement Python's MLPClassifier with 10 fold cross-validation using gridsearchCV function. Here is a chunk of my code:

parameters={
'learning_rate': ["constant", "invscaling", "adaptive"],
'hidden_layer_sizes': [(100,1), (100,2), (100,3)],
'alpha': [10.0 ** -np.arange(1, 7)],
'activation': ["logistic", "relu", "Tanh"]
}

clf
= gridSearchCV(estimator=MLPClassifier,param_grid=parameters,n_jobs=-1,verbose=2,cv=10)

Though,I am not sure if hidden_layer_sizes: [(100,1), (100,2), (100,3)] is correct. Here, I am trying to tune 'hidden layer size' & 'number of neurons'. I would like to give this 'tuple' parameter for hidden_layer_sizez: 1, 2, 3, and neurons: 10, 20, 30,...,100.

But I do not know if it is the correct way to do it. Therefore, I am choosing default neurons to be 100 in each layer.

Can anyone advise please?

$\endgroup$
3
  • $\begingroup$ I am very new to Python and was going through this post. My query is similar and response on setting up of Hidden layers helped a lot. However, I am unable to set up alpha in same way as mentioned above. Any help on this? I mean when I am setting up alpha as alpha': [10.0 ** -np.arange(1, 7)], it is giving me error. $\endgroup$
    – Abhik Jha
    Mar 4, 2018 at 13:14
  • $\begingroup$ the alpha parameter of the MLPClassifier is a scalar. [10.0 ** -np.arange(1, 7)], is a vector. Which works because it is passed to gridSearchCV which then passes each element of the vector to a new classifier. Have you set it up in the same way? $\endgroup$ Mar 4, 2018 at 14:03
  • $\begingroup$ 'hidden_layer_sizes': [x for x in product(range(1,100), range(1,3))] $\endgroup$ Sep 28, 2020 at 16:30

2 Answers 2

8
$\begingroup$

A tuple of the form $(i_1, i_2, i_3, ... , i_n)$ gives you a network with $n$ hidden layers, where $i_k$ gives you the number of neurons in the $k$th hidden layer.

If you want three hidden layers with $10,30$ and $20$ neurons, your tuple would need to look like $(10,30,20)$.

$(100,1)$ would mean that the second hidden layer only has one neuron.

$\endgroup$
4
  • $\begingroup$ Thanks for your reply, but I still do not get it. I would like to tune two things simultaneously; 'Number of layers ranging from 1 to 3', and 'Number of neurons in each layer ranging as 10, 20, 30, 40, 50, 100'. Can you please show in my above example code how to do it? Alternately, let's say I fix on 3 hidden layers. Now, I want to tune only neurons ranging as 10, 20, 30, 40, 50, 100 $\endgroup$
    – zx mnb
    Jun 16, 2017 at 18:04
  • 3
    $\begingroup$ you need to spell out all the combinations [(10,),...(100,),...(100,100,100)]. you can use itertools to generate them, e.g. [x for x in itertools.product((10,20,30,40,50,100),repeat=3)] if you really want all the combinations of of neurons for 3 hidden layers... $\endgroup$
    – oW_
    Jun 16, 2017 at 18:09
  • 1
    $\begingroup$ Thanks! I get it now what you mean, however, can you please provide a working example in my above code? $\endgroup$
    – zx mnb
    Jun 16, 2017 at 18:17
  • 1
    $\begingroup$ 'hidden_layer_sizes': [x for x in itertools.product((10,20,30,40,50,100),repeat=3)] ? $\endgroup$
    – oW_
    Jun 16, 2017 at 18:22
4
$\begingroup$

You can implement MLPClassifier with GridSearchCV in scikit-learn as follows (other parameters are also available):

GRID = [
    {'scaler': [StandardScaler()],
     'estimator': [MLPClassifier(random_state=RANDOM_SEED)],
     'estimator__solver': ['adam'],
     'estimator__learning_rate_init': [0.0001],
     'estimator__max_iter': [300],
     'estimator__hidden_layer_sizes': [(500, 400, 300, 200, 100), (400, 400, 400, 400, 400), (300, 300, 300, 300, 300), (200, 200, 200, 200, 200)],
     'estimator__activation': ['logistic', 'tanh', 'relu'],
     'estimator__alpha': [0.0001, 0.001, 0.005],
     'estimator__early_stopping': [True, False]
     }
]

PIPELINE = Pipeline([('scaler', None), ('estimator', MLPClassifier())])

You can then run GridSearch as the following:

grid_search = GridSearchCV(estimator=PIPELINE, param_grid=GRID, 
                            scoring=make_scorer(accuracy_score),# average='macro'), 
                            n_jobs=-1, cv=split, refit=True, verbose=1, 
                            return_train_score=False)

grid_search.fit(X, y)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.