# Hyperparameter optimization, ensembling instead of selecting with CV criteria

While burning CPUs performing a CV selection on a thin grid put on some hyperparameter space. I am using the $$$$scikit-learn' API, for which the end result is a single point on the hyperparameter space, for which the performance is the best according to chosen metric. It looks like a lot of information is trashed, we are probing the performance landscape of the hyperparameters and keeping a single point estimate. Information such as having many modes, local maximas, maybe far apart. An ensembler method could make good use of such information. Is there a theory built over ensembling a diversity of hyperparameter estimates?

• Well, there's whole world of optimization. Modern techniques go far beyond simple GA or PSO, though they're still considered computationally heavy. That's why they're used rarely for tuning of hyperparameters or network architecture search. – Piotr Rarus - Reinstate Monica Dec 13 '19 at 15:49

I think you are looking for hyperopts, Optuna and Gpopy for Hyperparameters search without burning much of CPUs.

• I have played with Bayesian optimizers before. I have found them very useful in some cases, and not so much in others. I regret that they are not parallelizable. I appreciate the suggestion, and it's always good to have them in the toolbox. – Learning is a mess Dec 17 '19 at 15:25

An excellent question, I think.

sklearn's hyperparameter searches actually don't keep any of the generated models, and instead refit a model on the best hyperparameter point at the end (optionally). Actually, since the performances are estimated using cross-validation, you don't ever generate any final models until the refit.

I'm not aware of any tools to do what you request, but it shouldn't be hard to generate one yourself. You get the cv_results_` table, and now want to select several high-performing points that are "far" from each other, then fit the model using each point. I guess with ensembling you might need some further training set then. The main difficulty seems to be in defining "far"?

• Thank you for the input. Indeed sklearn's API gives you access to all the parameters and their respective score value. I have been playing with that a bit to ensemble some models. I will try explaining my heuristic in an edit of my question. – Learning is a mess Dec 17 '19 at 15:27