2
$\begingroup$

I'm interested in trying to develop a framework for automatically tuning the hyperparameters of deep learning models and I'm looking for advice / references to resources that may be useful and suggestions/critiques on my proposed strategy below. I realise that experimentation may be the best way forward but I don't want to unnecessarily duplicate somebody else's work.

I know that a popular approach is to assume that the function of model error wrt. HPs is a bayesian process, the parameters of which are updated as new points are evaluated (ie. an expensive full training of the underlying deep learning model) so that a more rapid convergence on the optimal HPs can be achieved by quickly predicting better candidates for the next HPs to be evaluated than a simple grid search would be able to. Ilija Ilievski et al report more efficient convergence by using an RBF surrogate model (with pySOT) instead of bayesian https://github.com/ilija139/HORD.

Given that changes in some HPs (let's call them cheap) eg. ADAM initial learning rate and learning rate decay won't affect training time wheras doubling some HPs (let's call them expensive) eg. SGD batch size or total number of steps will double training time, does it make sense to approach this problem by trying to quickly refine cheap HPs whilst the object function is cheap(er) to evaluate and then allowing the object function to become more expensive gradually as the expensive HPs inflate?

One way of implementing this startegy might be to hold expensive HPs at relatively low values whilst refining cheap HPs and then allowing expensive HPs to grow and then iterating this process until some precondition is met, ie. model-error no longer goes down or processor-time budget is exceeded.

An alternative and possibly more 'organic' approach that I imagine would effectively achieve something similar would be to give each evaluation of the object function a certain processor time budget that would gradually increase. So when the object function over-ran its budget, it would be (gracefully) stopped and whatever error result that model gave, would be taken alongside whatever HPs were used (with number of steps being the actual number of steps taken, rather than what was requested). The idea being that this would pressure the HP surrogate to rapidly refine the cheap HPs initially whilst allowing the expensive HPs to be refined over time.

Does this make sense?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I like to start with an overfitting set of hyperparameters, train to convergence - usually zero training error - then start increasing the regularization until the test error starts declining. Time is saved by the fact that I never re-initialize, but jack up the regularization parameters in an already-fit network.

My regularization parameters are dropout rate and weight decay. I usually leave one of them fixed and move along a grid of the other. When trying for a given set of hyperparameters, I usually start with large batches (faster) and then move to small batches (deeper).

I also rely on ensembling, combining the prediction of 4 - 64 models trained analogously, maybe with different architectures

Ymmv, perhaps.

Improve this answer
$\endgroup$
2
  • $\begingroup$ thanks for the reply, I wonder if you could elaborate as to how starting by training an overfitted model and then introducing dropout saves time? Is it simply because you don't have to do all of those dropout calculations for the initial phase of object function evaluations? $\endgroup$
    – ezekiel
    Oct 21, 2017 at 11:09
  • $\begingroup$ It is because I only train one model. Rather than fitting several models, I fit a wiggly model and slowly make it more rigid. I prefer to leave dropout fixed and tune weight decay, usually $\endgroup$
    – generic_user
    Oct 21, 2017 at 12:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.