I recently read the LIPO blog post on the dlib blog: http://blog.dlib.net/2017/12/a-global-optimization-algorithm-worth.html

It mentions that it can be used for optimizing hyperparameters of eg metaheuristic algorithmsike simulated annealing or genetic algorithms.

I looked for info on how optimizing hyperparameters work in general and the Wikipedia page is the most informative I found but it doesn't answer my basic questions: https://en.m.wikipedia.org/wiki/Hyperparameter_optimization

My question is just: what is the basic idea for optimizing hyperparameters?

If I have some problem I'm trying to solve with simulated annealing, I know that the starting temperature and the cooldown rate are important in determining how well the algorithm does at finding a solution.

I know that I could completely run the algorithm with one set of parameters, modify one of the parameters, completely run it again, then reset the parameters and modify the other parameter and run it again. This could give me a numerical gradient that I could use to modify the parameters via gradient descent.

However... At this point I had to run the whole algorithm 3 times just to get a single modification of the hyperparameters.

I feel like I must be missing something obvious because optimizing the hyperparameters would take many many hundreds or thousands of times or more the cost of running the whole thing once, which doesn't seem useful at all. Can someone clue me in?


Hyperparameter optimization follows the same rules as model selection. Each set of hyperparameters effectively represents a different model you are considering, so the data you use to fit the model with some set of hyperparameters needs to be different from the data you use to evaluate which set of hyperparameters you want to ultimately use. A common approach for evaluating hyperparameters is nested cross-validation. This basically means that you need to treat hyperparameter selection as part of your model training process, and when you evaluate your model you evaluate the entire process front to back, i.e. treating hyperparameter tuning as a component of model training with respect to cross-validating over your training process. There's an excellent discussion of this in section 7.10.2 ("The Wrong and Right Way to Do Cross-validation") of Elements of Statistical Learning, which you can read online and/or download for free. The general idea is that if you're not careful, you can actually overfit to your evaluation data. Play with this demo to see for yourself.

But yes, your intuition is correct. Hyperparameter tuning is often very computationally expensive. One way people sometimes try to minimize this cost is by limiting the search space of feasible parameters to a small discrete set, e.g. grid search. Another approach is to use guassian processes or KDEs to approximate the cost surface in parameter space. You can even use a multi-armed bandit approach.

Frankly, you can really use pretty much any non-linear optimization technique for hyperparameter tuning, as long as you follow the appropriate cross-validation rules. The trick is figuring out bang-for-your-buck in terms of how much time/effort/compute you are willing to expend exploring parameters vs. the potential improvement to your model. Additionally, there are concerns about the limits of repeated cross-validation/holdout evaluations, but that's a whole other rabbit hole.

  • $\begingroup$ Ah ok, so tuning is more for quality, not so much for speed, sounds like. $\endgroup$ – Alan Wolfe Jan 3 '18 at 23:42
  • $\begingroup$ It can be both, but yeah: I'd say you should consider tuning a quality thing. "hyperparameters" basically describe the architecture of your model. In something like a NN, hyperparameters could be as small as picking the learning rate or as big as the number/type/dimension of hidden layers and kinds of activations to use. So they'll definitely have an impact on your training time, but when people talk about tuning they're generally more concerned with predictive performance than computational performance. That doesn't mean you can't incorporate training time into your tuning cost function. $\endgroup$ – David Marx Jan 4 '18 at 3:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.