An overview of the hyperparameter optimization process in scikit-learn is here.

Exhaustive grid search will find the optimal set of hyperparameters for a model. The downside is that exhaustive grid search is slow.

Random search is faster than grid search but has unnecessarily high variance.

There are also additional strategies in other packages, including scikit-optimize, auto-sklearn, and scikit-hyperband.

What is the most efficient (finding reasonably performant parameters quickly) method for hyperparameter optimization in scikit-learn?

Ideally, I would like working code examples with benchmarks.

  • $\begingroup$ I suspect the answer will depend a little on the model type. Did you have a specific one in mind? $\endgroup$ – Ben Reiniger Mar 13 at 20:24
  • $\begingroup$ Within scikit-learn you could also try scikit-hyperband. Sorry I don't have code to benchmark at the moment. Other methods exist however that are not implemented in scikit learn. $\endgroup$ – Ethan Mar 13 at 21:09
  • 1
    $\begingroup$ Hyperopt or using Bayesian Approach seems to dominate kaggle.. And obviously Experience thereafter as one can't really do it always :) $\endgroup$ – Aditya Mar 13 at 21:11
  • $\begingroup$ In scikit-learn, I generally use tree ensembles. Tree ensembles would be a good place to start given they tend to perform well and have many knobs to turn. $\endgroup$ – Brian Spiering Mar 13 at 21:18
  • $\begingroup$ @Ethan I have edited my question to add your excellent suggestion. I'm hesitant to jump on the latest bandwagon because new code is often buggy. I value working code over hype. $\endgroup$ – Brian Spiering Mar 13 at 21:24

You can take a look at auto-sklearn. That's an automated machine learning toolkit which is a direct extension of scikit-learn.

  • 1
    $\begingroup$ Very interesting. This uses Bayesian approach for hyper-parameter optimization under the hood @Aditya. $\endgroup$ – Esmailian Mar 13 at 23:37

Optimization isn't my field, but as far as I know, efficient and effective hyper-parameter optimization these days heavily revolves around building a surrogate model. As models increase in complexity, they become a more opaque black box. This is the case for deep neural nets and presumably complex trees as well. A surrogate model attempts to regress the underlying space within that black box. Based on a variety of sampling techniques, they probe the hyper-parameter space and attempt to build a function which represents the true underlying hyper-parameter space.

Bayesian optimization focuses on the surrogate model and how this model is constructed is crucial to BO. Also crucial to BO is choosing a good loss function.

I think performance between random search and Bayesian search varies from dataset to dataset, and model to model. Bergstra & Bengio (2012) made a strong argument for random search over grid search. Shahriari et al. (2016) make a strong case for BO. Model-based Hyperband strategies can potentially perform better than BO, especially for high-dimensions, however it is purely exploration, not exploitation. This can easily result in too early of stopping. However, there have been efforts to combine Hyperband and BO.

I've had good success scikit-optimize, despite there being quite a bit unimplemented. It's easy to prototype with and can easily interface with scikit-learn.

Bergstra, J., & Bengio, Y. (2012). Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb), 281-305.

Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & De Freitas, N. (2016). Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1), 148-175.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.