1
$\begingroup$

Are there methods to tune and train an xgboost model in an optimized time - when I tune paramaters and train the model it takes around 12 hours to execute?

I would like to run the solution 100 times with 100 seeds; my machine has 8 GB RAM and I can't buy a cloud solution.

Improve this question
$\endgroup$

3 Answers 3

Reset to default
3
$\begingroup$

Typically XGBoost does not require many parameters to be tuned to get good performances.

I would start with optimizing the n_estimators, max_depth and min_child_weight parameters only. This should already bring you close enough.

Another thing you can do to speed-up the process is to prefer Random Search over Grid Search, since in most cases is as or more effective.

You could also have a look at the LightGBM implementation, which is faster (and needs less memory) than XGBoost.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Another option is using a "smart" search algorithm - in packages like BayesianOptimazation, hyperopt, or skopt. $\endgroup$
    – bradS
    Commented Jul 8, 2019 at 10:32
  • 2
    $\begingroup$ I would advise against tuning n_estimators and would use the early stopping method instead. Also, if your data is of high dimension tuning stuff like colsample_bytree/level/node can be useful. $\endgroup$
    – aranglol
    Commented Jul 8, 2019 at 13:28
  • $\begingroup$ @aranglol what do you mean exactly by "high dimension"? Many features/inputs? Or does high dimension include rows as well? Would it also include a combo of how many categories a feature has? e.g. shoes_column=high-tops, cross-trainers, walking-shoes. $\endgroup$
    – Edison
    Commented Jun 30, 2022 at 2:28
  • $\begingroup$ @Edison I wrote this a while ago, but dimension always refers to the number of columns, not rows. In general, having a lot of rows is never bad from a pure statistical perspective. High dimension can also refer to the number of distinct categories as well, which is often referred to as the cardinality of a variable. Note that if you do one hot encoding for example (which is the standard way to deal with categorical variables like your shoe example), this implies that a categorical variable with a large # of categories will add a large number of columns, i.e. make your dimension "higher". $\endgroup$
    – aranglol
    Commented Jul 1, 2022 at 3:31
2
$\begingroup$

You can use less folds in cross validation, for example cv=2 or cv=3 in GridSearch to save time and memory.

I usually optimize a few parameters like: learning_rate, max_depth, min_child_weight and reg_alpha.

And since you mentioned you can't buy cloud solution, there is a free cloud called Google Colab https://colab.research.google.com/ which offers free RAM (I think up to 25 GB) and GPU. You can upload your ipython notebook and dataset or link it to your Google Drive. I use it for parameters tuning because it's faster than my low RAM computer.

Improve this answer
$\endgroup$
1
$\begingroup$

Another way of running hyperparameter optimisation is through Bayesian optimization. A package like skopt is already built to implement that in the context of xgboost and other scikit-learn predictors.

Check the documentation here: https://scikit-optimize.github.io/#skopt.BayesSearchCV

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.