1
$\begingroup$

When using Asynchronous Hyperparameter Optimization packages such as scikit optimize or hyperopt with cross validation (e.g., cv = 2 or 4) and setting the number of iteration to N (e.g., N=100), should I expect:

  1. Dependency between sequential iterations where the loss value improves sequentially (e.g., the optimized hyperparameters in iteration number 10 are better than the optimized hyperparameters generated in iteration number 9, etc.). In this case I should always select with the hyperparameters generated in the last iteration.

or

  1. Expect independency between iterations where after all 100 iterations are completed I should select the iteration with the smallest loss value.

If option a) is the right answer, that what does it mean if the best Hyperparameter are associated with iteration 50, does it mean that the data is not stable, or the loss function is ill-specified, and therefore, the hyperparameter optimization process outcome should not be trusted?

$\endgroup$
2
$\begingroup$

hyperopt proceeds sequentially, unless you let it use a parallel backend. Is N the max_evals parameter? Yes, you always want to select the hyperparams with the best validation loss. That's what it returns in the end. It may be that it found the best hyperparameters well before the final trial. It does not mean anything is wrong. It learns as it goes the distribution of the loss conditional on hyperparameters, and on purpose explores less-certain parts of the space that are most likely to yield improvement, but may not, especially towards the end of the search. This happens with entirely well-defined loss functions.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Yes, N is the max_evals parameter. In most of my cases the iteration with the best hyperparams (i.e., best validation loss) is never the last one. I am using n_jobs=-1 for the cross_validate function, is that is what you mean by 'parallel backend.' ? $\endgroup$ – thereandhere1 Mar 12 at 5:01
  • $\begingroup$ In both cases the framework should return the best (lowest loss) params / model, regardless. Yes you would not expect the models to be monotonically improving. $\endgroup$ – Sean Owen Mar 12 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.