Is there a rule of thumb for a sufficient number of trials for hyperparameter search

I am implementing a quite complicated Bayesian hyperparameter search in hyperopt library on a CNN.

Is there a rule of thumb for a "sufficient" number of trials? Perhaps based on the number of hyperparameters that constitute the search space?

• Nice question; about your your comment, how do you evaluate the possible number of params of the search space in case some of those params are continuous variables – German C M Jan 13 at 14:12
• Nice question too! This deserves a question by itself, I suggest you to write one because it'd be very helpful for the whole community. – Leevo Jan 13 at 15:29

1 Answer

In a pure random search, 60 points is often given as a rule of thumb, because provably with probability 95% such a search finds a hyperparameter combination in the top 5%.

However, that 5% is as a percent of the volume of the search space, so giving much-too-broad a search space, the best 5% might not be a fantastic score for the model. So it does seem to depend, not exactly on the number of hyperparameters, but on the volume of good combinations, which in turn indirectly depends on the number of hyperparameters.

Bayesian optimization should take fewer iterations (but not necessarily less time!) to reach an optimum than a pure random search, and in particular should rule out swaths of bad volume relatively quickly.

So, to conclude...no, I don't actually have a rule of thumb. 60 is probably a decent bet, given the tradeoff of the last two paragraphs; I'd go up to 100 if the training isn't too expensive. Also consider whether your package allows you to continue a Bayesian search: you could analyze the results so far (mostly, see if the last few iterations have clustered around some point, and how widely the objective function varies) to see if you want to proceed. Finally, note that scikit-optimize sets a default of 50, but there doesn't seem to be much behind that.

• Yeah, hyperopt allows to save past trials in a Trials() object, that you can then unpickle and restore for the next search session. Highly suggested btw – Leevo Jan 13 at 15:32
• Unfortunately my model is very heavy, one epoch takes almost 1 hour... it's going to be a pain even with a GPU laptop. I thought about using only a subset of Training data but I'm afraid of overfitting. – Leevo Jan 13 at 15:39