When to use BayesianSearchCV and how it works?

Question

Can somebody highlight when to use BayesianSearchCV and how it works? I have seen the implementation of same on kaggle and wanted to explore it further.

Below is the link where the implementation has been performed.

Kaggle link to Bayesian Optimization

Answer 1

As mentioned in that Kaggle notebook, you can use it pretty much as just a drop-in replacement for other search methods (grid or random). Bayesian searches still are random searches over a predefined search space/distribution, but now the algorithm pays attention to how well hyperparameter combinations perform, and will put more emphasis on high-performing areas.

There are several ways to do this. In BayesianSearchCV, a surrogate function keeps track of how the algorithm thinks hyperparameters will perform, with some uncertainty. Yet another function, an acquisition function, balances the desire to exploit high-performing areas with that to decrease uncertainty about unexplored areas (to avoid falling into a local optimum). A couple of nice descriptions:
http://neupy.com/2016/12/17/hyperparameter_optimization_for_neural_networks.html#bayesian-optimization
http://krasserm.github.io/2018/03/21/bayesian-optimization/

SciKit-Opt is specifically tailored for optimizing very expensive functions; it spends a fair amount of time updating its surrogate and acquisition functions and choosing the next hyperparameter combination to test. By doing so, it generally requires fewer iterations to get good results, but spends longer per iteration. This is probably a good tradeoff for, say, neural networks, but less clearly for XGBoost. But if you really need to eke out the last ounce of performance, it might be worth doing Bayesian over a purely random search.
https://github.com/scikit-optimize/scikit-optimize/issues/334#issuecomment-290358126
http://krasserm.github.io/2018/03/21/bayesian-optimization/
https://scikit-optimize.github.io/notebooks/bayesian-optimization.html

(The search begins with a number of randomly selected hyperparameter combinations, defaulting to 10 in skopt. So in that version of the Kaggle notebook, actually just a random search is being performed.)