I am using MLPClassifier from sklearn and I would like to tune it with GridSearchCV. But I don't know which set of values to include for hidden_layer_sizes, max_iter, activation, solver, etc. How can I determine the best combination to use in order to pass it to param_grid in GridSearchCV. How can I determine it?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
There is no go-to when it comes to hyperparameter selection for the GridSearchCV since each problem/dataset is different and GridSearchCV will be the one to decide which one/set of params is the best. Opting for a exhaustive search to try out all the parameter combinations are resource consuming and simply not a good practice.
I believe that you are trying to figure out which set of hyperparameter values will be best suited for your MLPClassifier, Generally you don't have to worry about selecting any specific hyperparameters to begin with but there are few things you can consider while doing so.
1. Pick arbitrary values:
Be cautious when using random values since it might not work all the time but try to have a range from which you want the hyperparams to be from.
Try to include values with a fixed interval lets say 2,5 10 or even 100. That's why most data scientists pick arbitrary value ends with a fixed intervals as their hyperparameter candidates which is a far better approach than including all values from 1 to 100 (just as an example).
2. Understanding different hyperparameters and cross-reference from others:
A good understanding about the hyperparams will be crucial in-order to help you include only the required parameters without trying out everything which can save some time.
For example, for solver you can use 'Sgd' or 'Adam' to begin with since they are one of the most used parameters and Adam is a go-to for most people including me (You can also read some research work to see how others does it as well on projects that are similar to yours).
Similarly for params like max-iter (Maximum number of iterations), it is not optimal to include values that are too high, Let's say your arbitrary choice was 50,000 (would result in 50,000 epochs). So it is important to have some clue about each param you want to include so you can avoid obvious traps and mishaps.
3. Computational Constraints:
Try to assess what system/devices you are working with because you don't want to punish your setup by entering extremely high values for params like max-iter, hidden layers etc.
It is generally a good practice to not have very low or very high values for certain params like max_iter and hidden_layer_sizes since low values may result in poor performance and extremely high values might put unnecessary computational stress on your device.
These above measures can indeed help you solve your problems when it comes to selecting hyperparams for grid search.
You can also read the Official documentation for more details on what each hyperparameter does and on how you should approach them.
