In your algorithms, when you use Gradient Boosting, do you prefer RandomSearchCV or GridSearchCV in order to optimize your hyperparameters ?
Thanks for sharing your experience.
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
I think it depends on the size of your multi dimensional grid. If it is small, then you can afford to be exhaustive and do a grid search. But if it is very large, and your computation time for a grid search extends too much, then definitely go to a random search. In fact, with random search one can explore larger regions than with grid search, and that is an advantage.
In any case, for hyperparameters search there are two keys:
Once you find the best regions, you can do either perform a more constrained there with either grid search or random search again.
Another option, which tends to work very well, is bayesian optimisation. Here the library that you use is important. In Python, after trying several which gave different problems, the best I found was skopt: https://scikit-optimize.github.io/