I have dataset to predict customers dropout(yes,no), with 5 numerical features and 2 categorical features. I have applied a scaler to the numerical data and transformed the categorical features into dummies variables, creating 29 features. My dataset has shape of 6552 rows and 34 features. What is the recommend approach to tune the parameters of XGBClassifier, since I created the model using default values, i.e., model=XGBClassifier()? Should I use a brute-force looping the values in some parameters until I find a optimal prediction value? In this case what is recommended?

  • $\begingroup$ Please go through the docs of xgboost... $\endgroup$
    – Aditya
    Mar 14 '18 at 10:40
  • $\begingroup$ Use Grid Search CV for range of values of parameters. scikit-learn.org/stable/modules/generated/… $\endgroup$
    – Ankit Seth
    Mar 14 '18 at 10:45
  • 2
    $\begingroup$ I would also suggest trying "Catboost" or "LightGBM" to have an idea what baseline performance accuracy you should expect from XGBoost. Especially Catboost is seen to give good first-time performance using default parameters, whereas XGBoost require lots of trial and error. ANd GridSearch often fails to be useful, and you end up tuning one parameter at a time! Usually you start with depth and try to overfit the training set, and add regularization next steps. There is also a bayesian optimization to explore parameter space (rather better than Grid), but I was not successful using it properly!! $\endgroup$ Mar 14 '18 at 12:25

There are three main techniques to tune up hyperparameters of any ML model, included XGBoost:

1) Grid search: you let your model run with different sets of hyperparameter, and select the best one between them. Packages like SKlearn have routines already implemented. But also in this case you have to pre-select the nodes of your grid search, i.e. which values have to be tried by the routine

2) Random search: similar to Grid Search, but you basically only choose the parameters boundaries, and the routine randomly try different sets of hyperparameters.

more informations about method 1 and 2 are here.

3) Bayesian optimization algorithms; this is the way I prefer. Basically this algorithms guesses the next set hyperparameter to try based on the results of the trials it already executed. An easy to use and powerful is SMAC.

  • $\begingroup$ Thanks for the suggestions. I will analyze it and try to apply your suggestions. Great :-) $\endgroup$ Mar 14 '18 at 15:35

When using GridSearchCV with XGBoost, be sure that you have the latest versions of XGBoost and SKLearn and take particular care with njobs!=1 explanation.

import xgboost as xgb
from sklearn.grid_search import GridSearchCV

xgb_model = xgb.XGBClassifier()
optimization_dict = {'max_depth': [2,4,6],
                     'n_estimators': [50,100,200]}

model = GridSearchCV(xgb_model, optimization_dict, 
                     scoring='accuracy', verbose=1)


Word of advice with XGBoost, careful with overfitting. In your particular situation, I suggest Catboost since it is optimized for categoricals.

  • $\begingroup$ Richard thanks for the example and the suggestions. I will try Catboost. Great $\endgroup$ Mar 14 '18 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.