While burning CPUs performing a CV selection on a thin grid put on some hyperparameter space. I am using the `scikit-learn' API, for which the end result is a single point on the hyperparameter space, for which the performance is the best according to chosen metric. It looks like a lot of information is trashed, we are probing the performance landscape of the hyperparameters and keeping a single point estimate. Information such as having many modes, local maximas, maybe far apart. An ensembler method could make good use of such information. Is there a theory built over ensembling a diversity of hyperparameter estimates?
An excellent question, I think.
sklearn's hyperparameter searches actually don't keep any of the generated models, and instead refit a model on the best hyperparameter point at the end (optionally). Actually, since the performances are estimated using cross-validation, you don't ever generate any final models until the refit.
I'm not aware of any tools to do what you request, but it shouldn't be hard to generate one yourself. You get the
cv_results_ table, and now want to select several high-performing points that are "far" from each other, then fit the model using each point. I guess with ensembling you might need some further training set then. The main difficulty seems to be in defining "far"?