2
$\begingroup$

Say that I have a dataset that contains 30 attributes which all of it is vital for my prediction and the dataset contains 500k rows. I would like to do a grid search for the best hyperparameters for the XGB model. How could i speed up the hyparameter search time as it would take a long time to find best parameters? would subset the data be useful?

$\endgroup$

4 Answers 4

2
$\begingroup$

First, you can optimize your function on a smaller set of random rows to reduce calculating time.

Then, you can optimize them on the first round, instead of the whole quantity of rounds. Generally speaking, what works well on the initial rounds, works well for the other ones.

However, I recommend testing on the complete rounds and data set at the end with a few scenarios, to ensure that it is actually a good optimal setting.

Finally, use fast GPUs. If they are too expensive, cloud services like Paperspace have good ones.

$\endgroup$
2
$\begingroup$

You can use sklearn GridSearchCV, which has a parameter called n_jobs, and according to the documentation

n_jobs : int, default=None

Number of jobs to run in parallel. None means 1 unless in a joblib.parallel_backend context. -1 means using all processors. See Glossary for more details.

So by setting n_jobs=-1 you'll be running several experiments at the same time, and the overall time to get the best hyperparameters will be reduced.

However, there are better techniques than grid search, for example, bayesian optimization. With bayesian optimization, you let the information of past rounds guide where to look for the best hyperparameters, so with less iterations you can get better results than in grid search. Optuna is a python library that allows you to do bayesian optimization, and also allows you to easily parallelize the tuning.

$\endgroup$
1
$\begingroup$

this is a pretty new way of optimized grid cv https://scikit-learn.org/stable/modules/grid_search.html#successive-halving-user-guide

$\endgroup$
1
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Jun 30, 2022 at 16:12
0
$\begingroup$

Yes: take subsets of your data.

Given you have 500K rows, one approach is to randomly sample two blocks of 5K rows. Run grid search (see other answers for suggestions for that) on your first block. Then repeat the experiment on your second block. If your chosen hyperparameters are the same the second time, smile, and use them. If different, choose a bigger sample, and compare your two candidates.

There are loads of variations on this basic idea. You could use the first two runs to narrow down the hyperparameters, but then increase the sample sizes to fine-tune them.

I said random samples above, but also consider taking representative samples. This would mean making sure the distribution of each of your 30 attributes in your sample roughly matches its distribution in the entire 500K rows. Also, if your dataset is unbalanced, or has lots of missing values, this can require some careful thinking about what you are trying to achieve.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.