Estimating Length of Hyperband Trials in Advance

Question

I would like to use the (Keras/Tensorflow) hyperband tuning algorithm more than the Keras random search, for instance, when testing hyperparameters.

With random search I can set max trials and get a really rough guess of how long it will go on (probably by an order of magnitude uncertainty from max_trials*epochs).

With hyperband I don't know how long it will take, or if I'm setting a search that's going to be really limited. Is there a way to make sense of e.g. max_epochs=10, factor=3, hyperband_iterations=10 (already knowing what they mean) to make a guess?

I don't quite understand how many possible solutions hyperband will search on its first go, second, etc. and the rate of increase of increase to calculate a loose and fast max_trials*epochs equivalent.

This could also be useful to people wanting to use/compare these algorithms (if searches on other models were done with one algorithm, then you may want to be fair by giving/not over/under-searching other models if you do it in another method). (?)

Answer 1

This is estimated in the documentation, under the hyperband_iterations parameter description:

hyperband_iterations: Integer, at least 1, the number of times to iterate over the full Hyperband algorithm. One iteration will run approximately max_epochs * (math.log(max_epochs, factor) ** 2) cumulative epochs across all trials. It is recommended to set this to as high a value as is within your resource budget. Defaults to 1.

So the answer is approximately $$ \texttt{hyperband_iterations} \cdot \texttt{max_epochs}\cdot \left(\log_{\texttt{factor}}(\texttt{max_epochs})\right)^2 $$

You may be interested in the referenced paper for an explanation of where that formula comes from, in particular Table 1. There are logarithmically many brackets, each of which has at most logarithmically many halvings, and the total epochs used in each step of the halving is roughly the maximum epochs.