I have dataset to predict customers dropout(yes,no), with 5 numerical features and 2 categorical features. I have applied a scaler to the numerical data and transformed the categorical features into dummies variables, creating 29 features. My dataset has shape of 6552 rows and 34 features. What is the recommend approach to tune the parameters of XGBClassifier, since I created the model using default values, i.e., model=XGBClassifier()? Should I use a brute-force looping the values in some parameters until I find a optimal prediction value? In this case what is recommended?
$\begingroup$
$\endgroup$
3
asked Mar 14, 2018 at 10:31
-
$\begingroup$ Please go through the docs of xgboost... $\endgroup$– AdityaCommented Mar 14, 2018 at 10:40
-
$\begingroup$ Use Grid Search CV for range of values of parameters. scikit-learn.org/stable/modules/generated/… $\endgroup$– Ankit SethCommented Mar 14, 2018 at 10:45
-
2$\begingroup$ I would also suggest trying "Catboost" or "LightGBM" to have an idea what baseline performance accuracy you should expect from XGBoost. Especially Catboost is seen to give good first-time performance using default parameters, whereas XGBoost require lots of trial and error. ANd GridSearch often fails to be useful, and you end up tuning one parameter at a time! Usually you start with depth and try to overfit the training set, and add regularization next steps. There is also a bayesian optimization to explore parameter space (rather better than Grid), but I was not successful using it properly!! $\endgroup$– TwinPenguinsCommented Mar 14, 2018 at 12:25
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
1
There are three main techniques to tune up hyperparameters of any ML model, included XGBoost:
1) Grid search: you let your model run with different sets of hyperparameter, and select the best one between them. Packages like SKlearn have routines already implemented. But also in this case you have to pre-select the nodes of your grid search, i.e. which values have to be tried by the routine
2) Random search: similar to Grid Search, but you basically only choose the parameters boundaries, and the routine randomly try different sets of hyperparameters.
more informations about method 1 and 2 are here.
3) Bayesian optimization algorithms; this is the way I prefer. Basically this algorithms guesses the next set hyperparameter to try based on the results of the trials it already executed. An easy to use and powerful is SMAC.
answered Mar 14, 2018 at 14:24
-
$\begingroup$ Thanks for the suggestions. I will analyze it and try to apply your suggestions. Great :-) $\endgroup$ Commented Mar 14, 2018 at 15:35
$\begingroup$
$\endgroup$
1
When using GridSearchCV with XGBoost, be sure that you have the latest versions of XGBoost and SKLearn and take particular care with njobs!=1 explanation.
import xgboost as xgb
from sklearn.grid_search import GridSearchCV
xgb_model = xgb.XGBClassifier()
optimization_dict = {'max_depth': [2,4,6],
'n_estimators': [50,100,200]}
model = GridSearchCV(xgb_model, optimization_dict,
scoring='accuracy', verbose=1)
model.fit(X,y)
print(model.best_score_)
print(model.best_params_)
Word of advice with XGBoost, careful with overfitting. In your particular situation, I suggest Catboost since it is optimized for categoricals.
-
$\begingroup$ Richard thanks for the example and the suggestions. I will try Catboost. Great $\endgroup$ Commented Mar 14, 2018 at 16:06
$\begingroup$
$\endgroup$
XGBoost's defaults are pretty good. I'd suggest trying a few extremes (increase the number of iterations by alot, for example) to see if it makes much of a difference. If you do see big changes (for me it was only ~2% so I stopped) then try gridsearch. XGboost trains very quickly.
Tree classifiers like this are great in that normalization isn't needed.
I like the feature_score tracking you can do (not sure how in python, check the docs) if you want to see which of your features is contributing most to the result. This is especially useful if some metrics are expensive to compute, so you can see what to bother with, but can also be useful in a business sense (e.g. maybe how long they've been a customer or time since last sale doesn't contribute at all).
