11
$\begingroup$

An overview of the hyperparameter optimization process in scikit-learn is here.

Exhaustive grid search will find the optimal set of hyperparameters for a model. The downside is that exhaustive grid search is slow.

Random search is faster than grid search but has unnecessarily high variance.

There are also additional strategies in other packages, including scikit-optimize, auto-sklearn, and scikit-hyperband.

What is the most efficient (finding reasonably performant parameters quickly) method for hyperparameter optimization in scikit-learn?

Ideally, I would like working code examples with benchmarks.

Improve this question
$\endgroup$
7
  • $\begingroup$ I suspect the answer will depend a little on the model type. Did you have a specific one in mind? $\endgroup$
    – Ben Reiniger ♦
    Commented Mar 13, 2019 at 20:24
  • $\begingroup$ Within scikit-learn you could also try scikit-hyperband. Sorry I don't have code to benchmark at the moment. Other methods exist however that are not implemented in scikit learn. $\endgroup$
    – Ethan
    Commented Mar 13, 2019 at 21:09
  • 1
    $\begingroup$ Hyperopt or using Bayesian Approach seems to dominate kaggle.. And obviously Experience thereafter as one can't really do it always :) $\endgroup$
    – Aditya
    Commented Mar 13, 2019 at 21:11
  • $\begingroup$ In scikit-learn, I generally use tree ensembles. Tree ensembles would be a good place to start given they tend to perform well and have many knobs to turn. $\endgroup$
    – Brian Spiering
    Commented Mar 13, 2019 at 21:18
  • 1
    $\begingroup$ Interestingly, some recent benchmarks have show that random x2 can outperform some of the "fancier" methods. I can link to a really good article/paper for you to read. Also, if I find some time next week I may be able to answer your question more fully. $\endgroup$
    – Ethan
    Commented Mar 19, 2019 at 0:16

2 Answers 2

Reset to default
7
$\begingroup$

Optimization isn't my field, but as far as I know, efficient and effective hyper-parameter optimization these days heavily revolves around building a surrogate model. As models increase in complexity, they become a more opaque black box. This is the case for deep neural nets and presumably complex trees as well. A surrogate model attempts to regress the underlying space within that black box. Based on a variety of sampling techniques, they probe the hyper-parameter space and attempt to build a function which represents the true underlying hyper-parameter space.

Bayesian optimization focuses on the surrogate model and how this model is constructed is crucial to BO. Also crucial to BO is choosing a good loss function.

I think performance between random search and Bayesian search varies from dataset to dataset, and model to model. Bergstra & Bengio (2012) made a strong argument for random search over grid search. Shahriari et al. (2016) make a strong case for BO. Model-based Hyperband strategies can potentially perform better than BO, especially for high-dimensions, however it is purely exploration, not exploitation. This can easily result in too early of stopping. However, there have been efforts to combine Hyperband and BO.

I've had good success scikit-optimize, despite there being quite a bit unimplemented. It's easy to prototype with and can easily interface with scikit-learn.

Bergstra, J., & Bengio, Y. (2012). Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb), 281-305.

Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & De Freitas, N. (2016). Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1), 148-175.

Improve this answer
$\endgroup$
3
+50
$\begingroup$

You can take a look at auto-sklearn. That's an automated machine learning toolkit which is a direct extension of scikit-learn.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ Very interesting. This uses Bayesian approach for hyper-parameter optimization under the hood @Aditya. $\endgroup$
    – Esmailian
    Commented Mar 13, 2019 at 23:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.