0
$\begingroup$

The hyperparameter search is computationally expensive. I am wondering if one can tune the hyperparameters independently: tune one hyperparameter for a fixed value of other hyperparameters. For example, let's say we have two hyperparameters, A and B. We search for best value of A at fixed random B, then we search for best value of B at fixed best value of A.

This make sense if the the other hyperparameters does not interfere with the ordering of the validation loss for the hyperparameter we want to tune. In that sense, the number of units and the number of layers cannot be tuned independently. According to the Y. Bengio's paper (link), at some point the mini-batch size can be tuned independently (page 9, right column, The Mini-Batch Size).

But what about the other ones? Learning rate, activation function, dropout, ... which one can be tuned independently?

Improve this question
$\endgroup$
6
  • $\begingroup$ Could you please refer where in the paper, author mentions that "the mini-batch size can be tuned independently." $\endgroup$ – BlackCurrant May 17 '20 at 12:33
  • $\begingroup$ @BlackCurrant I added the page number to the post. $\endgroup$ – New Developer May 17 '20 at 12:40
  • $\begingroup$ By Looking at that, one can't entirely say that author is talking about that we can fix training size and then tune other hyper parameters. its still says" after the other hyper-parameters (except learning rate) have been selected". Then once batch size is selected it optimizing other parameters even more.I would not come to the conclusion that A and B are totally independent here . $\endgroup$ – BlackCurrant May 17 '20 at 12:45
  • $\begingroup$ @BlackCurrant Good point. Well, it is kind of confusing for me. As I understand it, it says that order of tuning should be like, first other hyperparameters, then Batch size and at the end learning rate, then reoptimize both Batch size and learning rate. Then at fixed Batch size reoptimize the others. Then at some point it can be considered as independent. $\endgroup$ – New Developer May 17 '20 at 13:14
  • $\begingroup$ My point is that although the test error for every combination of A and B can be different, still it might be the case that in a predetermined grid search the ordering (not the values) of test errors for A can be independent of the value of B. $\endgroup$ – New Developer May 17 '20 at 13:21
0
$\begingroup$

Your question is indeed a little bit broad, but I will try to explain to give you an overview on the importance and on some specific topics.

Indeed, multiple hyper-parameters exist in the context of deep learning. At the same time, just like Andrew Ng mentions in his courses, some are of an bigger importance than the others.

For instance, if you see that your training progresses very slowly (i.e. your convergence is relatively slow), you may want to fine-tune your learning rate.

Learning rate is a quintessential example of hyper-parameter that is more important than the number of neurons in a FullyConnected layer or to change the Dropout rate on a layer from 0.3 to 0.5.

At the same time, there exist two well known techniques for hyper-parameter search: grid search and random search. While the former behaves exactly like you explained (keeping the values of N-1 hyper-parameters fixed and iterating over some specific values of the N hyper-parameter), the random search has proven to have a better positive impact, as it slightly modifies all your hyper-parameters after a search step; although it may not be intuitive at first sight, this can yield better and earlier results than the grid search.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the response. What I explained is different from a brute-force grid search. Let's say we have two hyperparameters: A and B, each of 10 values. In brute-force grid search, every combination is considered, in total 100 computations. Whereas in what I explained, we search for best value of A at fixed random B (10 computations), then we search for best value of B at fixed best value of A (10 computations). So, in this case the number of computations is smaller (20). But it works if A and B does not affect the ordering of each other. $\endgroup$ – New Developer May 17 '20 at 11:37
  • $\begingroup$ Thank you, I understood better now what you are saying, it is now clearer. I hope that the comment apart from this idea helped you understand better. $\endgroup$ – Timbus Calin May 17 '20 at 12:02
0
$\begingroup$

As per the paper,The Author has concluded-

"the wisdom distilled here should be taken as a guideline, to be tried and challenged,

not as a practice set in stone

. The practice summarized here, coupled with the increase in available computing power, now allows researchers to train neural networks on a scale that is far beyond what was possible at the time of the first edition of this book, helping to move us closer to artificial intelligence"

So we can't say that its a practice and we can keep everything fixed and only tune learning rate first and then keep learning rate fixed and tune weights.It doesn't even seems reasonable knowing how GD/errors are calculated and weights are updated.

By this-

""the mini-batch size can be tuned independently.""

It seems he meant the we can tune parameters on this fixed batch and take these parameters as guidelines and tune even further. " Hope it helps.

On the other note, I don't see any tables/comparison in the papers which has results of proposed techniques. May be you should contact the author about the understanding.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.