Hyperparameter optimization, ensembling instead of selecting with CV criteria

While burning CPUs performing a CV selection on a thin grid put on some hyperparameter space. I am using the $$$$scikit-learn' API, for which the end result is a single point on the hyperparameter space, for which the performance is the best according to chosen metric. It looks like a lot of information is trashed, we are probing the performance landscape of the hyperparameters and keeping a single point estimate. Information such as having many modes, local maximas, maybe far apart. An ensembler method could make good use of such information. Is there a theory built over ensembling a diversity of hyperparameter estimates?

• Well, there's whole world of optimization. Modern techniques go far beyond simple GA or PSO, though they're still considered computationally heavy. That's why they're used rarely for tuning of hyperparameters or network architecture search. Dec 13, 2019 at 15:49

I'm not aware of any tools to do what you request, but it shouldn't be hard to generate one yourself. You get the cv_results_` table, and now want to select several high-performing points that are "far" from each other, then fit the model using each point. I guess with ensembling you might need some further training set then. The main difficulty seems to be in defining "far"?