Hyper parameters tuning XGBClassifier

Question

I am working on a highly imbalanced dataset for a competition.

The training data shape is : (166573, 14)

train['outcome'].value_counts()

0    159730 
1      6843

I am using XGBClassifier for building model and the only parameter I manually set is scale_pos_weight : 23.34 (0 value counts / 1 value counts)

and it's giving around 82% under AUC metric.

I guess I can get much accuracy if I hypertune all other parameters.

XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
       colsample_bytree=1, gamma=0, learning_rate=0.1, max_delta_step=0,
       max_depth=3, min_child_weight=1, missing=None, n_estimators=100,
       n_jobs=1, nthread=None, objective='binary:logistic', random_state=0,
       reg_alpha=0, reg_lambda=1, scale_pos_weight=23.4, seed=None,
       silent=True, subsample=1)

I tried GridSearchCV but it's taking a lot of time to complete on my local machine and I am not able to get any result back.

clf = XGBClassifier()
grid = GridSearchCV(clf,
                    params, n_jobs=-1,
                    scoring="roc_auc",
                    cv=3)

grid.fit(X_train, y_train)
print("Best: %f using %s" % (grid.best_score_, grid.best_params_))

What others parameters should I target to tune considering higly imbalanced dataset and how to run it so that I can actually get some results back?

Answer 1

1)Random search if often better than grid https://www.analyticsindiamag.com/why-is-random-search-better-than-grid-search-for-machine-learning/ Try:

#clf = RandomizedSearchCV(clf ,param_distributions = params, cv=kfold, scoring="accuracy", n_jobs= 10, verbose = 1)

2) For unbalanced data set:

• Resampling: undersampling or oversampling
• create new data points

https://towardsdatascience.com/methods-for-dealing-with-imbalanced-data-5b761be45a18

Answer 2

I think you are tackling 2 different problems here:

1. Imbalanced dataset
2. Hyperparameter optimization for XGBoost

There are many techniques for dealing with Imbalanced datasets, one of it could be adding higher weights to your small class or another way could be resampling your data giving more chance to the small class.

For XGBoost I suggest fixing the learning rate so that the early stopping number of trees goes to around 300 and then dealing with the number of trees and the min child weight first, those are the most important parameters.