0
$\begingroup$

Are there methods to tune and train an xgboost model in an optimized time - when I tune paramaters and train the model it takes around 12 hours to execute?

I would like to run the solution 100 times with 100 seeds; my machine has 8 GB RAM and I can't buy a cloud solution.

$\endgroup$
0
$\begingroup$

Typically XGBoost does not require many parameters to be tuned to get good performances.

I would start with optimizing the n_estimators, max_depth and min_child_weight parameters only. This should already bring you close enough.

Another thing you can do to speed-up the process is to prefer Random Search over Grid Search, since in most cases is as or more effective.

You could also have a look at the LightGBM implementation, which is faster (and needs less memory) than XGBoost.

$\endgroup$
  • $\begingroup$ Another option is using a "smart" search algorithm - in packages like BayesianOptimazation, hyperopt, or skopt. $\endgroup$ – bradS Jul 8 at 10:32
  • 1
    $\begingroup$ I would advise against tuning n_estimators and would use the early stopping method instead. Also, if your data is of high dimension tuning stuff like colsample_bytree/level/node can be useful. $\endgroup$ – aranglol Jul 8 at 13:28
0
$\begingroup$

Another way of running hyperparameter optimisation is through Bayesian optimization. A package like skopt is already built to implement that in the context of xgboost and other scikit-learn predictors.

Check the documentation here: https://scikit-optimize.github.io/#skopt.BayesSearchCV

$\endgroup$
0
$\begingroup$

You can use less folds in cross validation, for example cv=2 or cv=3 in GridSearch to save time and memory.

I usually optimize a few parameters like: learning_rate, max_depth, min_child_weight and reg_alpha.

And since you mentioned you can't buy cloud solution, there is a free cloud called Google Colab https://colab.research.google.com/ which offers free RAM (I think up to 25 GB) and GPU. You can upload your ipython notebook and dataset or link it to your Google Drive. I use it for parameters tuning because it's faster than my low RAM computer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.